Ordering Across the Device Boundary: DMA, MMIO, and Posted Writes

A device driver writes a descriptor into memory, then writes the descriptor’s address into a doorbell register mapped in the device. The device reads the descriptor and finds garbage. The store to memory happened before the store to the register in program order, the core’s own ordering rules would keep the two in that order for other threads, and yet the device saw the register write without the memory write.

The boundary between a processor and a device breaks every assumption in the earlier parts of this cluster at once. There is no coherence domain spanning the two, the ordering point they share is a bus or a port rather than a cache hierarchy, and the observation that a store completed does not mean it will ever be seen by the device through the path the device uses.

Two paths to memory, one of them not coherent

The root cause is that the processor and the device reach memory by different routes. The processor’s stores go through its caches and are made visible to other processors by the coherence protocol. The device’s reads go through an I/O path, which may bypass the caches entirely or may snoop them, depending on the platform.

When the device’s path snoops the caches, the coherence domain is extended to include it, and ordering behaviour resembles the intra-processor case. This is common on integrated platforms where a coherent interconnect includes device ports. When the device’s path does not snoop, the processor must explicitly push dirty data to memory before the device is told to read it, and the fence that guarantees this is not an ordering fence but a cache maintenance operation followed by an ordering fence.

/* Descriptor handoff: writes must reach memory before the doorbell. */
struct desc { uint64_t addr; uint32_t len; uint32_t flags; };

static void submit(struct desc *d, uint64_t addr, uint32_t len)
{
    d->addr  = addr;
    d->len   = len;
    d->flags = DESC_READY;

    /* Order the descriptor stores before the register write below. */
    __atomic_thread_fence(__ATOMIC_SEQ_CST);

    /* Doorbell: MMIO store, not an ordinary cached store. */
    writel((uint32_t)(uintptr_t)d, doorbell);
}

The fence orders the descriptor stores against the doorbell write. On a platform where the device’s path does not snoop, an additional operation is required to clean the descriptor’s cache lines, and the correct sequence is clean-then-fence-then-write. The order matters: cleaning first without the fence allows the doorbell write to overtake the cleaning, and fencing first without the clean leaves the data in a cache the device cannot read.

Posted writes and why completion is not delivery

The second mechanism is specific to the I/O fabric. A write to a device register is typically posted: the root complex accepts it and acknowledges completion to the processor before the device has seen it. From the processor’s perspective the store retired; from the fabric’s perspective it is in flight.

That means two consecutive writes to device registers are ordered by the fabric but neither may have arrived when the processor proceeds. A subsequent read from the device, however, is not posted: the processor must wait for the device to respond. The read therefore acts as an implicit ordering point, because the fabric cannot return the read’s response until all earlier posted writes to that device have been delivered. This is the classical rule that a read flushes posted writes, and it is why drivers sometimes force a read-back of a register as a cheap barrier.

Cheap is the operative word. A read-back costs a fabric round trip, hundreds of nanoseconds on a peripheral bus, and it is not the same guarantee as an explicit ordering barrier because it orders only against the endpoint that answers the read. The portable form is the architecture’s I/O ordering barrier, which orders prior writes against later writes without a round trip on implementations that can enforce the order in the fabric.

The distinction between the two directions is worth stating plainly. Going processor-to-device, the hazard is a write that has not arrived. Going device-to-processor, the hazard is data the device wrote that the processor has not observed, either because it is still in the fabric or because the processor has a stale cached copy.

The receive side: why a device’s writes need a barrier too

The mirror-image problem appears when the device writes to memory and then signals the processor, typically by writing a status field or raising an interrupt. The processor’s interrupt handler reads the status, then reads the data. If the processor’s load of the data is satisfied from a stale cache line, or if the two reads are reordered, the handler operates on old contents.

Two things are needed. First, the device’s writes must be visible before the notification is; this is a property of the device and the fabric, and it is specified by the device’s programming interface rather than by the processor. Second, the processor must not read the data before it has read the notification, and must not use a cached value that predates the device’s write.

The second requirement is where the interrupt boundary is involved. An interrupt is not an ordering operation in the memory model; it is a control transfer. The fence that establishes the order is the one in the handler, placed between the read of the status and the read of the data. Platforms whose device path does not snoop also require the cache to be invalidated for the data region before reading it, which is a maintenance operation distinct from the fence.

Direct memory access with a non-coherent path therefore imposes a symmetric discipline: on transmit, clean and fence before notifying; on receive, fence and invalidate after being notified and before reading. Omitting either half produces corruption that depends on cache state, so it appears intermittently and correlates with unrelated code changing the cache’s contents.

What the memory model says about MMIO

A register mapped into the address space is not ordinary memory, and the language memory model says nothing useful about it. The standard’s atomic operations apply to objects in ordinary memory; a device register is outside that model, and an atomic access to it has undefined meaning because the hardware does not implement the ordinary read-modify-write behaviour on the device side.

Two practical rules follow. Accesses to device registers must be performed through operations the implementation defines for that purpose, and those operations must be opaque to the compiler: it must not merge two register reads into one, split a write, or reorder them against each other. Whether that opacity comes from a volatile qualifier, from a compiler builtin, or from a dedicated accessor depends on the language and the toolchain, and the chosen form must also carry the ordering the device requires.

The second rule is that a volatile qualifier is not sufficient for ordering. It prevents some compiler transformations on the access itself and provides no ordering against other memory, and no ordering at the hardware level. Code that relies on volatile alone for a handoff has a data race in the language model and an unordered sequence in the hardware model, and it works only when the platform happens to be strongly ordered and the compiler happens not to reorder. This is the same trap as in Compiler Reordering and the Contract It Must Not Break, with the additional hazard that the observer is a device rather than a thread.

Multi-socket paths and the scope problem

On a machine with several processor sockets, the path from a core to a device may traverse links between sockets, and the ordering guarantees that hold within a socket do not automatically extend across those links. The barriers that exist at the narrower scope are sufficient for ordering within it and insufficient for the wider one, and the device may be attached to a different socket from the one running the driver.

This is the same layering as the domain-qualified barriers on weakly ordered architectures, described in Propagation, Cumulativity, and Why Weak Models Need Transitive Visibility, but with an additional twist: the device is not a participant in the coherence protocol unless the platform makes it one. A barrier that orders within an inner shareable domain orders nothing for a device outside it, and the correct barrier is the one whose scope covers the path to the device.

The practical consequences for driver code are a small set of rules that are worth stating as rules because the failure mode is intermittent corruption rather than a crash.

Every handoff to a device is a release operation on the processor side and requires an ordering operation between the data and the notification, at a scope covering the path to the device.

Every notification from a device is an acquire operation on the processor side and requires an ordering operation between the observation of the notification and the read of the data.

Non-coherent paths require cache maintenance around those operations, in the order clean-fence-notify and observe-fence-invalidate.

A read-back of a register orders posted writes to the same endpoint but costs a fabric round trip; the architecture’s I/O ordering barrier is the general form and does not require a response.

None of these rules is implied by the processor’s own memory model, which is why driver code cannot be reasoned about with the model that applies to threads. The model ends at the boundary, and the ordering discipline at the boundary is a separate contract between the processor, the fabric, and the device.

Why the boundary stays hard

The reason these rules persist in every platform specification, rather than being absorbed into a single uniform model, is that the device side is not under the processor architect’s control. A device may buffer writes, coalesce reads, reorder internally, and complete a transfer long after the driver believes it has begun. The only ordering the processor can enforce is the ordering of its own accesses as they reach the fabric, and everything beyond that must be specified by the device’s interface.

That is also why the strongest tool at the boundary is not a barrier but a protocol: a descriptor with an ownership bit that only one side writes at a time, a status field the device updates only after the data is in memory, an interrupt raised only after the status is visible. The barriers make the protocol work; they cannot substitute for it. A driver that relies on barriers without a protocol has ordered its own accesses correctly and still cannot know what the device did with them.