Mapping Physical RAM Across Isolated Processes
When two processes communicate over a pipe or socket, every message traverses the kernel twice. POSIX shared memory bypasses the kernel on the data path entirely: once two processes map the same shared memory object into their respective virtual address spaces, they read and write the exact same physical RAM frames directly from user space.
Yet shared memory introduces a subtle trap that catches almost every systems programmer at least once: code that works flawlessly with pointers in a multi-threaded program will immediately segfault when moved to shared memory. Let’s examine how the POSIX shared memory bridge works under the hood, why virtual addresses break across process boundaries, and how to design position-independent shared data structures in modern C.
Setting Up the Shared Memory Bridge
Establishing a shared memory region in POSIX requires three system calls: shm_open, ftruncate, and mmap.
| |
Each step fulfills a distinct contract with the operating system kernel:
shm_open: Creates or opens a POSIX shared memory object backed by a RAM-resident filesystem (tmpfsat/dev/shmon Linux) and returns a file descriptorfd.ftruncate: A newly created shared memory object has a length of zero bytes. Mapping a zero-length object and dereferencing the resulting pointer raises a hardware bus error (SIGBUS) because no backing physical pages exist.ftruncateresizes the object tosizeof(SharedRing).mmapwithMAP_SHARED: Instructs the kernel to wire the object’s physical frames into the calling process’s page tables. Note that we can immediatelyclose(fd)aftermmapreturnsβthe virtual memory mapping holds its own kernel reference to the underlying backing object.
Why Pointers Break Across Process Boundaries
Look at SharedRing above: int32_t slots[RING_CAPACITY] is embedded directly inside the struct as a fixed-size array. Suppose instead that a developer wanted a dynamically sized buffer and wrote:
| |
If Process A creates the shared segment and initializes the buffer with ring->slots = malloc(cap * sizeof(int32_t)), Process A can read and write ring->slots[i] without issue. The instant Process B attaches to the shared memory and evaluates ring->slots[0], Process B crashes with SIGSEGV.
Two Separate Mechanisms Cause This Failure
1. Heap Allocations Are Process-Private
When Process A calls malloc(), the memory allocator (brk or anonymous mmap) allocates pages inside Process A’s private heap, not inside /dev/shm. Writing the pointer into ring->slots only copies an 8-byte integerβthe virtual address 0x55a100402010βinto shared memory. In Process B’s page tables, virtual address 0x55a100402010 either is unmapped (SIGSEGV) or points to unrelated private data inside Process B.
2. Even Pointers Inside the Shared Region Change Address (ASLR)
What if Process A doesn’t use malloc(), and instead points ring->slots to another address inside the shared memory region itself?
It still breaks! When Process A and Process B call mmap(nullptr, ...), the kernel is free to place the shared mapping at different virtual base addresses in each process (especially under Address Space Layout Randomization, ASLR):
| |
In Process A, the array inside the shared region lives at 0x7f8e42c00010. In Process B, that exact same physical memory lives at 0x7f1100a00010. If Process A writes the absolute pointer 0x7f8e42c00010 into the shared header, Process B will dereference 0x7f8e42c00010βan unmapped address in Process B!
Designing Position-Independent Shared Structures
To make a data structure safe across processes, every reference inside the shared region must be position-independent: its meaning cannot depend on the virtual base address returned by mmap. Systems programmers use three idiomatic patterns in C to achieve this.
Pattern 1: Embedded Fixed Arrays (Compile-Time Sizing)
When the maximum capacity is known at compile time, embed the array directly inside the struct (as in SharedRing above). When either process accesses ring->slots[i], the compiler emits an address calculation relative to that process’s own local ring pointer:
$$\text{Address} = \text{local_base} + \texttt{offsetof(SharedRing, slots)} + i \times \texttt{sizeof(int32_t)}.$$
Pattern 2: C99 Flexible Array Members (Runtime Sizing)
What if the capacity $N$ is only known at runtime when the producer starts? Use a Flexible Array Member (int32_t slots[];) as the final field of the struct, and size the shm_open segment dynamically:
| |
A consumer attaching later simply maps the first sizeof(DynamicSharedRing) bytes to read ring->capacity (or checks fstat(fd, &st) to get st.st_size), and maps the full regionβwith zero pointers stored in shared memory.
Pattern 3: Relative Offset Pointers (Linked Structures & Arenas)
What if you are building a dynamic linked list, B-tree, or slab allocator inside shared memory? Instead of storing an absolute virtual address (Node *next), store a byte offset (ptrdiff_t or uint64_t) relative to either the base of the shared memory segment or the address of the pointer field itself (“self-relative pointer”):
| |
Because the distance in bytes between two locations inside the same contiguous mmap region is identical in every process, (const char *)rp + rp->offset always resolves to the exact local virtual address in whichever process evaluates it.