Shared Memory Pitfalls and Best Practices - Sort Out Synchronization, Visibility, Lifetime, ABI, and Security First

The short answer first

Shared memory is “IPC that reduces copying, but pushes the responsibility for consistency onto the application.”

• It is fast when you are moving large data inside a single machine
• For small control messages, pipes or sockets are easier
• Being visible and being safe to read are different problems
• volatile is not a synchronization spell
• Drop in raw pointers, std::string, or std::mutex and you will cry later

When shared memory fits, and when it doesn’t

Four things to decide first

1. Separate the control plane from the data plane - data goes through shared memory, notifications go through events / pipes / sockets
2. Narrow the concurrency model - SPSC (one-to-one) is the easiest
3. Decide ownership and lifetime - who creates it, who destroys it
4. Decide the ABI and version - layout, type sizes, version number

Common pitfalls

1. Not synchronizing

“We are looking at the same memory, so once I write it, the other side can read it” is wrong. There is no guarantee that the reader sees it at the right moment, in the right granularity, or in the right order. Always combine it with a separate synchronization primitive such as a mutex, semaphore, or event.

2. Trying to fix things with volatile

volatile guarantees neither atomicity nor mutual exclusion. A design that busy-loops on volatile bool ready wastes CPU and is prone to picking up intermediate states.

3. Letting readers see intermediate states

When you publish a record made of multiple fields, a reader can end up seeing “the new length combined with the old payload.” Countermeasures:

• Protect with a mutex
• Use a double buffer and switch the active buffer index
• Use a ring buffer with state/sequence per slot

4. Dropping pointers or complex objects in directly

Raw pointers, HANDLE, std::string, std::vector, and std::mutex cannot cross process boundaries. Hold references as offsets from the base address.

5. Breaking the ABI

Shared memory is a binary contract, not a source-code contract. Differences in int/long size, the representation of bool, 32-bit vs. 64-bit, alignment, and padding all matter. Use fixed-width integers (uint32_t and friends) and verify with static_assert(sizeof(...)).

6. Initialization races

Assuming “the creator should have initialized it” will burn you. Put a state field (INITIALIZING / READY / BROKEN) in the leading header, let only the creator initialize it, and make joiners wait for READY.

7. Ignoring crash recovery

What happens if the writer dies mid-update? At minimum, carry a generation number, the last committed sequence, a heartbeat, and a dirty/clean flag.

8. Stuffing notifications into shared memory too

while (!ready) Sleep(1); wastes CPU. Push notifications out to a waitable primitive (event, semaphore).

9. Underestimating names and permissions

On Windows there are Global\ and Local\ namespaces. Creating something in Global\ from outside session 0 requires SeCreateGlobalPrivilege. Object names share a namespace across events, mutexes, and semaphores, so watch out for collisions.

Best practices

Put a fixed header at the front

typedef struct SharedHeader {
    uint32_t magic;          // reject the wrong thing or uninitialized
    uint16_t abi_version;    // reject layout differences
    uint16_t header_size;
    uint32_t state;          // 0=initializing, 1=ready, 2=broken
    uint64_t total_size;
    uint64_t generation;     // detect re-creation
    uint64_t heartbeat_ns;   // liveness
    uint64_t payload_offset;
    uint64_t payload_size;
    uint8_t  reserved[64];   // escape hatch for future extensions
} SharedHeader;

Other best practices

• Hold references as offsets - resolve as base + offset and add range checks
• Narrow the concurrency model - start with an SPSC ring buffer or a double buffer
• Make the commit protocol explicit - decide “from which moment is it OK to read”
• Fix the size per generation - prefer creating a new generation and switching over rather than resize-in-place
• Build in observability - last-update time, drop count, version mismatch count, heartbeat
• Write the failure-path tests first - writer kill, reader stall, version mismatch, insufficient permissions

APIs to use on Windows and POSIX

C#’s MemoryMappedFile is essentially a wrapper around Windows file mapping. The basic rules are the same.

Wrap-up

The real essence of shared memory is not “speed” but a transfer of responsibility. In exchange for fewer copies, you take on synchronization, ABI, recovery, and permissions yourself.

Build your first one like this:

• An SPSC ring buffer or a double buffer
• A fixed leading header (magic, version, state, generation)
• Offset-based references
• Notifications on a separate channel
• Failure-path tests in place