Choosing the Unit of Concurrency
Every concurrent system on UNIX begins with a structural trade-off: should workers execute as threads sharing a single address space, or as independent processes communicating across kernel-enforced boundaries?
Multi-threading is often treated as the default choice because sharing a pointer across threads costs nothing. Yet foundational systemsβPostgreSQL, Chromium’s renderer sandbox, Nginx, and OpenSSHβrely heavily on multi-process architectures. Understanding why requires looking at what a process actually isolates, how the hardware enforces that isolation, and what happens in the kernel when two processes need to exchange data.
Address Space Isolation vs. Shared Execution
To the kernel’s scheduler, both processes and threads are runnable execution contexts (struct task_struct on Linux). The distinction lies entirely in resource ownership:
- A process owns an independent virtual address space (backed by its own top-level page table loaded into the CPU’s
CR3/TTBR0register), a file descriptor table, signal handlers, and security credentials. - A thread (
pthread) is an execution context that shares its parent process’s page table, heap, global.data/.bsssegments, and open file descriptors, maintaining only a private stack, register state, and thread-local storage (errno).
| |
When a process calls fork(), the kernel duplicates its page table entries as read-only and marks them for Copy-on-Write (CoW). Both processes share the underlying physical RAM frames until one of them writes to a page, trapping into the kernel’s page-fault handler to allocate a private physical copy.
The Architectural Trade-Off
| Dimension | Threads (pthread) | Processes (fork / Independent Binaries) |
|---|---|---|
| Fault Containment | None: A single out-of-bounds write or SIGSEGV in one thread terminates the entire process. | Strict: If a worker process crashes, the OS reclaims its descriptors and memory while sibling processes continue unaffected. |
| Deployment Coupling | Tightly bound to a single binary and lifecycle. | Independent binaries (e.g., a daemon and a CLI client) can start, restart, or upgrade separately. |
| Privilege Separation | All threads share the same UID, GID, and address space. | Workers can drop privileges (setuid), enter chroot/namespaces, or apply restrictive seccomp filters. |
| Communication Cost | $O(1)$ pointer passing in user space (requires synchronization). | Requires explicit Inter-Process Communication (IPC). |
When fault containment, privilege separation, or independent lifecycles matter, separate processes are mandatoryβand we need a mechanism to cross the address-space wall.
The Three Families of POSIX IPC
POSIX.1b defines three distinct IPC models, each trading kernel involvement against memory bandwidth:
1. Kernel-Buffered Message Passing (Pipes, FIFOs, Message Queues)
In message-passing channels (pipe, mkfifo, mq_open), processes share no memory. Sending $N$ bytes requires two system calls and two memory copies:
write()/mq_send()copies $N$ bytes from the sender’s user-space buffer into a kernel ring buffer.read()/mq_receive()copies those $N$ bytes from the kernel buffer into the receiver’s user-space buffer.
Message passing is naturally serialized by the kernel and immune to cross-process memory corruption, making it ideal for control signals and moderate byte streams. However, for high-throughput telemetry or large shared data structures, copying megabytes twice through the kernel on every transaction quickly saturates memory bandwidth.
2. Shared Memory (shm_open + mmap)
POSIX shared memory eliminates data-path system calls and kernel copies altogether. By instructing the kernel to map the same physical RAM frames into the page tables of multiple processes (MAP_SHARED), any store instruction executed by Process A becomes immediately visible to load instructions in Process B at native L1/L2/L3 cache speeds:
| |
3. Cross-Process Synchronization (sem_open / sem_init)
Shared memory provides raw memory access, but zero synchronization. Because two processes can read and write mapped addresses simultaneously on different CPU cores, uncoordinated writes cause data races and torn reads.
To coordinate access across processes, POSIX provides semaphores (<semaphore.h>):
- Named Semaphores (
sem_open): Exposed in the filesystem namespace (/dev/shm/sem.*), allowing unrelated processes to acquire the same synchronization primitive by name. - Process-Shared Memory Semaphores (
sem_init): Embedded directly inside ammap’d sharedstructby passingpshared = 1.
(Note: Standard pthread_mutex_t and pthread_cond_t primitives can also work across processes if initialized inside shared memory with PTHREAD_PROCESS_SHARED and PTHREAD_MUTEX_ROBUST).
Kernel Persistence and Resource Hygiene
A crucial operational difference separates POSIX IPC objects from standard file descriptors or heap allocations: kernel persistence.
When a process terminatesβwhether normally via exit(0) or fatally via SIGKILLβthe kernel automatically closes its open file descriptors and unmaps its virtual memory. However, named shared memory segments (shm_open) and named semaphores (sem_open) survive process termination. They remain allocated in the kernel’s tmpfs mount (/dev/shm on Linux) until explicitly removed with shm_unlink() and sem_unlink(), or until the machine reboots.
Consequently, production IPC code follows two defensive rules:
- Minimize the Lifetime of Names: If the creator process knows its peers have already attached, it can unlink the name early (just like unlinking a temporary file while keeping its descriptor open), or register an
atexit()/SIGINThandler to unlink on shutdown. - Design Self-Contained Shared Layouts: Everything mapped into shared memory must obey strict value-semantics layout rulesβbecause as we will see next, virtual addresses themselves do not survive the crossing from one process to another.