Memory-Mapped Registers and the Ordering Rules Behind a Store

A device driver is mostly a program that writes small integers to addresses that are not memory. The integers are commands: start a transfer, set a mode, acknowledge an interrupt. The addresses are decoded not by DRAM but by a device on the other side of an interconnect. Everything about this arrangement looks like a memory access at the instruction level and behaves unlike one at the system level, and the gap between those two views is the source of most driver bugs that survive to production.

How a register becomes an address

At configuration time the firmware or the operating system walks each device’s configuration space and reads the base address registers. A base address register describes a range of addresses the device wants to claim. The host writes all-ones into the register, reads back what the device returns, and from the pattern of zeros learns the size and alignment the device requires. A device that returns zeros in the low bits is asking for a naturally aligned, power-of-two-sized window; the number of immovable low bits gives the size directly. A 64-bit capable device consumes two adjacent base address registers and encodes its type in the upper half of the first.

The host then assigns a physical address range to each register, writes the base back, and enables memory decoding in the command register. From that moment the device claims any access the host routes into its window. The window sizes are quantized: a device needing a handful of 32-bit registers still occupies a range whose size is the next power of two large enough to hold them, because the decode compares address bits against a mask. That quantization is why a device with five scattered registers may reserve a window far larger than the registers themselves, and why a system with many devices can run out of MMIO space rather than memory.

Two properties of the window matter for correctness. First, the addresses are not cached: the platform marks the range uncacheable so that every load reaches the device. Second, the range is not ordinary memory, so the compiler and the processor must be told not to reorder or coalesce accesses to it. In C the access is usually written through a volatile pointer, which stops the compiler from eliding or hoisting the access, and the ordering that volatile does not provide is supplied by architecture-specific barriers.

Posted writes and the meaning of “complete”

A write to a device register does not wait for the device. The transaction travels up the hierarchy as a posted write: each bridge forwards it and considers its obligation discharged, and the originating core retires the store as soon as the write leaves the core’s own ordering domain. No completion returns. The store instruction retires, the next instruction executes, and the device may not have seen the value yet.

This is a deliberate performance decision. Requiring a round trip for every write would make a register update cost a bus latency instead of a store issue slot, and register writes are often issued in bursts — a descriptor address, a length, a doorbell — where the round trips would serialize for no benefit. The cost is that “the write has been issued” and “the device has observed the write” are different events, and the gap between them is unbounded in principle.

Reads are not posted. A read must return data, so it carries a completion, and the core stalls until the completion arrives. That asymmetry is the basis for the one portable ordering trick available: after a sequence of writes that must be observed in order, read back a register from the same device. The read cannot complete until every earlier posted write to that device has been delivered, because the interconnect maintains order between posted writes and the subsequent read completion for the same requester and the same traffic class.

#include <stdint.h>

struct ctrl {
    volatile uint32_t command;   /* write-only in practice */
    volatile uint32_t status;    /* device updates this */
    volatile uint32_t doorbell;
};

static inline void flush_writes(struct ctrl *c)
{
    /* the load is not used; it exists to force earlier posted writes to land */
    (void)c->status;
    __asm__ __volatile__("" ::: "memory");  /* keep the compiler from removing it */
}

void start(struct ctrl *c, uint32_t desc, uint32_t len)
{
    c->command = len;      /* program the length first */
    c->doorbell = desc;    /* then point the device at the descriptor */
    flush_writes(c);
}

The dummy load is not a substitute for a full barrier when the ordering requirement spans other observers. It orders the requester’s writes with respect to that device. If the write must be visible to another observer — a peer device reading the same memory, or a CPU in another coherence domain — the required primitive is a system-scope fence, which is a stronger and more expensive operation than a read-back.

Read-modify-write hazards

Registers often mix host-owned and device-owned bits in one word. A status register may hold a device-maintained error count in some bits and host-writable enables in others. Writing the whole word back to change one bit destroys whatever the device put in the rest, and if the device updates the field between the read and the write, the write publishes stale values.

Devices handle this in one of three documented ways. Some provide a write-one-to-clear convention, in which writing a one to a bit clears it and writing zero leaves it alone, so a single store with the right bit set is safe and no read is needed. Some separate host-writable and device-written fields into different registers, which removes the hazard by construction and is the better design. Some offer atomic set and clear aliases at distinct addresses within the window, so a store to the alias performs the operation in the device without a read at all. A driver that reads a status word, modifies a bit, and writes it back is relying on none of these and is usually wrong.

The same hazard appears with 64-bit counters on a 32-bit access path. Reading the low half and then the high half can straddle a carry. Devices that expose such counters usually provide a latch mechanism — a read of the low half captures the high half, or a separate shadow register holds a consistent pair — and the driver must follow the documented sequence rather than two independent loads.

Width, alignment, and what the device actually decodes

A device may decode only certain access sizes. A register block designed for 32-bit accesses may ignore a 16-bit store or, worse, may accept it and place the bytes in the wrong lanes. Byte enables travel with the transaction, and a device that does not implement them drops the write silently. The practical rule is that the access size in the driver must match the size the register map specifies, which is why register definitions in drivers are typed precisely rather than as generic words.

Alignment is the second constraint. A naturally aligned 32-bit access is one transaction; a misaligned access may be split by the core into two, and two posted writes to adjacent addresses can arrive in either order at the device. A device that treats its register file as a stream of commands will then execute them out of order. Splitting also breaks the read-back guarantee, since the second half of a split write may still be in flight when the read returns.

The write-combining trap

Because MMIO writes are expensive per transaction, some ranges are marked write-combining: the core buffers stores and merges adjacent ones into a single larger transaction. This is a large win for framebuffer and bulk data paths, where hundreds of adjacent words become a handful of cache-line-sized writes.

It is a correctness hazard for registers. Writes to adjacent addresses can be merged into one wider transaction, and a device that expects a sequence of commands at consecutive addresses may see them arrive as a transaction whose size and shape it does not decode. Ordering between a write-combined buffer and a later read is also weaker: the buffer must be drained, which happens on a fence or when the buffer fills, and a plain read-back does not necessarily drain it. Ranges containing control registers are therefore mapped uncached rather than write-combining, and the mapping attribute is part of the device’s contract rather than a performance preference.

Where the abstraction leaks

Three failure modes account for a large share of MMIO bugs. A store is issued and the driver immediately frees the structure it described, because the store retired and the driver assumed the device had consumed it — the classic use-after-free that a read-back would have prevented. A driver polls a status bit without a bound and hangs when the device is in a state the bit never leaves. And a driver writes a whole register to change one bit, clearing an interrupt that another part of the driver was waiting on.

The mitigations are structural rather than clever. Ordering requirements belong next to the code that depends on them, not scattered across the driver. Register definitions belong in one place with their widths and access conventions written down. And the handoff of any buffer to the device belongs with the machinery that decides when the device is done with it, which is the subject of Descriptor Rings and the Producer-Consumer Contract and, for the memory the device reads, of Device DMA and the IOMMU Boundary.