Posted Writes and the Ordering Rules a Device Must Respect

A memory write over PCIe is a posted transaction: the requester sends the packet and considers the operation issued, with no acknowledgement of any kind. This is the single most consequential property of the protocol for driver authors, because it means a write that has been issued is not a write that has arrived. Everything else in the ordering rules follows from the need to make that fire-and-forget behaviour usable: some transactions must be allowed to overtake others or the fabric deadlocks, and some must not or software breaks.

Why writes are posted at all

The alternative, a write that waits for an acknowledgement, costs a full round trip per store. On a link with a few hundred nanoseconds of latency, a device that acknowledged every descriptor write would be limited to a few million writes per second no matter how wide the link, and the acknowledgement traffic would consume bandwidth in the opposite direction. Posting removes the round trip from the critical path: the write is buffered at the receiver, the sender moves on, and the buffer drains at whatever rate the destination can absorb.

The cost is that the sender has no idea when the write became visible. There is no completion for a posted write, no error reported to the sender if the receiver’s buffer overflows (flow control credits prevent that), and no way to ask. A write to an address that no bridge claims is discarded at the point where it cannot be routed, and no completion comes back to the sender; the receiving hierarchy reports an Unsupported Request through its error-reporting registers rather than to the device that issued the write. On PCI the same case produced a master abort; on PCIe the failure is reported locally and the requester’s only evidence is a read-back.

Posted writes also include message-signalled interrupts. An MSI is a memory write to an address that the interrupt controller decodes, so it is subject to the same ordering rules as any other write. The specification requires MSI and MSI-X transactions to be sent with the relaxed-ordering attribute clear, which keeps an interrupt behind the writes that preceded it within its traffic class. The remaining hazard follows from the difference between what those rules guarantee and what they do not.

The three classes and the rules between them

Every transaction belongs to one of three classes: posted requests (memory writes, messages), non-posted requests (reads, configuration accesses, I/O writes), and completions (the responses to non-posted requests). The ordering rules are stated as which class may pass which, and they exist to satisfy two requirements that pull in opposite directions.

The deadlock requirement says that a posted write must be able to pass a non-posted request. Consider a device that issues a read to a slow target and then a write to a different target behind it in the same queue. If the write cannot pass the read, the write waits for the read to complete; if the read’s completion is itself blocked behind the write at some point in the fabric, nothing progresses. PCIe resolves this by allowing posted writes to pass non-posted requests, which is why a switch can forward a write while a read is still outstanding.

The consistency requirement says that a non-posted request must not pass an earlier posted write. A read issued after a write must observe that write, or at least must not be serviced before it. Without this rule, a driver that writes a descriptor and then reads a status register would be reading a state that predates its own write. The default behaviour is therefore that reads do not pass writes.

Between posted writes the default is strong ordering. A posted request must not pass an earlier posted request, so a sequence of writes issued by one function reaches the fabric in the order it was issued. This is what makes the data-then-doorbell sequence safe on the link: the doorbell cannot overtake the descriptor writes that precede it, provided both carry the same traffic class. The cost is head-of-line blocking, because a switch must hold a later write behind an earlier one whose destination is full, and the specification relieves it in two ways. A posted write may be marked with the relaxed-ordering attribute, which permits it to pass an earlier posted write; a posted write marked with ID-based ordering may pass an earlier one when the requester IDs differ. Both attributes are opt-in and default to clear. Write combining is a separate matter: the specification prohibits the root complex and switches from combining memory writes.

The attributes are the escape hatch, and they are set per request rather than per device. Relaxed ordering on a posted write, or on a non-posted request with data, permits it to pass an earlier posted write. ID-based ordering permits a request to pass an earlier posted request when the two come from different requester IDs. Both are opt-in and default to clear, and setting either is a statement that the software or device does not depend on the earlier transaction being ordered ahead — a statement that is easy to make and hard to keep true when queue depths and traffic classes change at runtime.

The interrupt hazard

The classic sequence is the data-then-doorbell sequence. A device finishes a DMA transfer, writes a completion descriptor, and then signals the host by writing to its MSI address. Both the descriptor write and the MSI are posted writes, and within a single traffic class the second cannot pass the first, so the fabric does not reorder them. The guarantee stops at the fabric boundary: the MSI is consumed by the interrupt controller, the descriptor is consumed by memory, and the host decides when the interrupt reaches a core and when the descriptor becomes visible to it. A device that needs the write accepted before it signals therefore does not rely on the two posted writes alone. The same failure appears when the two writes carry different traffic classes, because the ordering rules say nothing across classes.

The portable fix is to insert a non-posted operation between the two writes. A read cannot pass an earlier posted write, so a device reads back a location it just wrote — its own register, or the descriptor it placed — and only then sends the interrupt. The read’s completion is ordered after the write, and the device waits for that completion before it issues the MSI, so the signal cannot be sent before the write was accepted. The sequencing is a control dependency rather than a bus-ordering rule, because a posted request is permitted to pass a non-posted request. The read costs a round trip, which is why high-throughput devices batch: complete many descriptors, read back once, then signal once.

The same reasoning applies to a driver that writes a doorbell register to start a device operation. The doorbell is a posted write; if the driver then immediately polls a status register, the poll is a read and cannot pass the doorbell, so the device will not service the poll before the doorbell. But if the driver writes the doorbell and then waits on an interrupt, the interrupt may arrive before the device has seen the doorbell: the two travel in opposite directions on the link, and the ordering rules say nothing about opposite directions. The driver must tolerate a spurious wakeup. The asymmetry — reads are safe after writes in the same direction, interrupts are not — is the practical rule to carry away.

Completion boundaries and split reads

A read of more than a small amount of data is not answered by one completion. The protocol defines a read completion boundary for the root complex, either 64 or 128 bytes, and a memory read that crosses a boundary may be answered by several completions, each tagged with the same requester and tag and each carrying a different portion of the data. The receiver must reassemble them in order, and the boundary exists so that a target with a narrow internal data path can begin returning data before the whole request has been fetched.

This matters because a requester’s completion buffer must be sized for the worst case, and because the number of completions for one read is not fixed by the request length. A 256-byte read with a 64-byte boundary may come back as four completions or as one, depending on the target’s buffering. A driver or device that assumes one completion per read works until it meets a target that splits, and then either drops data or hangs waiting for a completion count that will never arrive.

The status field inside a completion distinguishes a successful completion, an unsupported request, and a completer abort. A read to an address outside every bridge’s window returns an unsupported request with no data. For configuration reads, the requester synthesises all ones so that probing software sees the conventional empty-slot value; for a memory read there is no such convention, and a device that treats the completion status as an afterthought will use 0xFFFFFFFF and interpret it as a register value.

Flow control, and the ordering the credits impose

Posted writes consume flow control credits from the receiver’s posted-request buffer, and the credits are the mechanism that makes posting safe: a sender cannot overflow a receiver because it cannot send more than the receiver has advertised. The credit accounting is separate per class and split into header and data credits, which means a device that exhausts data credits for posted writes can still send header-only messages, and a device that exhausts header credits stops entirely even with data credit available.

The practical consequence is that a burst of writes to a device whose receive buffer is small will be throttled by credit return latency, not by link bandwidth. A receiver that returns credits slowly limits the sender to the window size divided by the round trip, exactly as a sliding window limits a transport protocol. Devices with deep buffers advertise large credits; devices that must handle a full-rate stream without stalling are built with enough buffer to cover the link’s bandwidth-delay product.

Because credit return is itself a data link layer event and not a transaction, it is not subject to the ordering rules above. A credit returning does not imply the corresponding write has reached memory, only that the receiver’s buffer has space. Conflating the two produces a driver that thinks a write is complete because the sender stopped being throttled.

What the rules do not guarantee

Three things are commonly assumed and not provided. First, no guarantee exists that a posted write from a device is visible to a core that reads the same address immediately afterwards. PCIe orders the device’s requests against each other; it does not tell a core when the write will reach memory or when the core’s cached copy will be invalidated. A core polling a flag without a synchronising event can read the old value. The read-back orders the device’s own traffic; it does not synchronise the device with a core.

Second, ordering is not the same as reaching the final destination. A read that is ordered after a posted write cannot be serviced before that write, so a read-back of the written location proves the write reached that location. A read of some other location, such as a status register in the device, proves only that the write was accepted into the fabric ahead of the read, not that it has been applied where it is going. A device that needs the stronger fact reads the written location, or waits for a platform-level completion such as a fence or an acknowledged interrupt.

Third, ordering does not cross traffic classes or directions. Two writes with different traffic-class labels have no ordering relationship, and a write in one direction is not ordered against a write in the other. A device that puts its descriptor writes in one class and its interrupt in another has removed the guarantee it was relying on, and the same mistake appears when a driver assumes that a device’s downstream write and its upstream interrupt are ordered against each other.

The link-level machinery that carries these transactions, including the sequence numbers that make delivery reliable but say nothing about ordering, is described in PCIe lanes and packet layers. Where a device’s writes are translated before they reach memory is the subject of DMA addressing and the IOMMU, and the topology that decides how many hops a write crosses is covered in PCIe topology and BDF enumeration.