PCIe Lanes and the Three Layers That Carry a Transaction

A PCIe link is not a bus. It is a serial, full-duplex, point-to-point connection, and every operation that a shared bus performed with wires and timing must be performed by a protocol instead. The protocol is layered so that each layer can be reasoned about in isolation: the transaction layer defines what a request means, the data link layer makes delivery reliable across a single hop, and the physical layer moves symbols across lanes and recovers a clock from them. Understanding a single memory read means following one packet down through all three.

A read becomes a packet

A host reads a device register by issuing a memory read to a bus address that the host bridge has routed to a link. The transaction layer builds a request packet: a header of three or four double-words (12 or 16 bytes), an optional payload for writes, and an optional end-to-end CRC. The header carries a type field, a requester ID, a tag, and the target address. For a 32-bit address the header is three double-words; a 64-bit address needs four, which is why a 64-bit read costs four extra bytes of overhead on every request.

The tag is the mechanism that makes a split-transaction protocol work without waiting. A requester may have multiple reads outstanding, and each carries a distinct tag; the completion that comes back names the requester ID and tag it belongs to, so responses can arrive out of order with respect to requests and still be matched. The number of tags a function supports is a capability: the base tag field is five bits, giving 32 outstanding requests per function, and an extended-tag capability widens it to eight bits, with the sixth generation extending it to fourteen in flit mode. This is the direct replacement for the shared bus’s wait states: instead of holding the bus while a device fetches data, the requester releases the link and is notified later.

Routing is by address, not by device selection. A memory request’s address is compared against the base and limit registers of each bridge on the path; a configuration request is routed by bus number, device number, and function number; a message is routed by a code that identifies the recipient type. The same three mechanisms cover everything, which is why a switch needs no per-transaction command decode and can forward on header fields alone.

The data link layer’s sequence numbers

A serial link at multi-gigabit rates will see bit errors, and a transaction layer that assumed reliable delivery would have to expose retries to software. The data link layer instead guarantees reliable delivery across one hop. Every transaction layer packet is wrapped in a data link layer packet: a two-byte sequence number in front, and a four-byte link CRC at the end. The receiver checks the CRC, checks that the sequence number is the next expected one, and acknowledges accordingly.

The acknowledgement is not per packet. The receiver maintains a counter of packets accepted and periodically returns an ACK or NAK with that count; a NAK causes the transmitter to replay everything from the last acknowledged sequence number, because the transmitter keeps a replay buffer of unacknowledged packets. The buffer is sized in units of packets and its depth is a property of the implementation, not something the two ends negotiate; what is advertised is the flow-control credit count, which bounds how many packets the receiver’s buffers can hold. The replay buffer still bounds throughput on a long-latency link in the way a TCP window does: bandwidth-delay product must fit inside the buffer or the link idles waiting for ACKs.

Replay is the reason a corrupt packet does not become a software error, and also the reason a link that persistently fails CRC will retry indefinitely until the physical layer declares the link down. There is no forward error correction at this layer in the original protocol; a packet either passes CRC or is replayed. Later generations add FEC at the physical layer for the same reason that high-speed serial links generally need it, which changes the error model from “detect and replay” to “correct most, replay the residue”.

Lanes, symbols, and the encoding overhead

The physical layer turns bytes into symbols and spreads them across lanes. A link of width N distributes consecutive bytes round-robin over the N lanes, so a x4 link moves four bytes per symbol time and a x16 moves sixteen. Width is negotiated at training time and can be reduced if lanes fail, which is why a x8 slot can end up running x4 or x1 and why bandwidth expectations derived from slot size alone are often wrong.

Encoding overhead differs by generation and is the single easiest number to get wrong. Generation 1 and 2 use 8b/10b: every eight bits of payload become ten bits on the wire, so twenty percent of raw bandwidth is spent on encoding. A Gen2 lane runs at 5 GT/s and spends a fifth of the raw rate on encoding, leaving 4 Gb/s of payload, which is 500 MB/s per lane per direction. Generation 3 raises the raw rate to 8 GT/s and switches to 128b/130b, where a two-bit sync header is attached to each 128-bit block (about 1.5 percent overhead), so a lane carries roughly 985 MB/s. The jump from Gen2 to Gen3 is therefore a little better than the raw 1.6x rate increase suggests, because the encoding tax fell from 20 percent to under 2 percent.

The receiver recovers its clock from the transitions in the incoming stream, which is why the encoding must guarantee a bounded run length without transitions. A long run of identical bits would let the receiver’s clock drift; the 8b/10b code guarantees a transition within a small number of symbols, and the 128b/130b scheme relies on the scrambler plus the sync header for the same purpose. This is also why a link cannot simply be idled: an idle must be encoded as a defined sequence of symbols, the electrical idle ordered set, so that the receiver’s clock recovery and the link state machine stay synchronised.

Ordering, flow control, and what the layers do not do

The transaction layer also owns flow control. Credits are advertised per buffer class — posted requests, non-posted requests, and completions, each split into header and data credits — and a transmitter may send a packet only if the receiver has enough credits of the right class. Credits are returned as buffers drain, so the mechanism is a per-class credit window rather than a per-packet permission. This is why a device that stops draining its receive buffers will stall a specific traffic class rather than the whole link, and why a memory write can pass a memory read that is blocked on non-posted credits.

Ordering rules are the other half of the transaction layer’s contract. By default a posted request must not pass an earlier posted request, and a read must not pass an earlier posted request, so a write posted by a device cannot be overtaken by a later read from the same device to the same address. A posted request must be able to pass a non-posted request, which is what prevents deadlock when a read is blocked. Completions must stay ordered with respect to other completions for the same requester and tag. The asymmetry is deliberate: allowing a write to pass a read keeps the fabric moving, and forbidding a read to pass a write keeps producer-consumer ordering intact. A device that needs the opposite guarantee, a write that must not overtake an earlier read, has no request attribute for it and must wait for the read’s completion before issuing the write.

What the layers do not provide is end-to-end reliability. The link CRC protects a single hop; a packet corrupted inside a switch’s buffer and forwarded with a fresh CRC would pass. The optional end-to-end CRC in the transaction layer header covers that case, and it is a capability that must be enabled at both ends — a link with ECRC disabled on one side silently carries no protection. Similarly, the protocol has no notion of a device being absent: a request to a function that has been removed returns an unsupported request completion, and it is the driver’s responsibility to have quiesced traffic before removal. The hot-plug model depends on the link and the slot controller, not on the packet layer.

Where the latency goes

A single register read over PCIe crosses all three layers twice. The request is built and transmitted, the switch forwards it after a header decode, the endpoint’s transaction layer receives it, its data link layer acknowledges, the completion is built and sent back, and the requester’s completion buffer matches it to the outstanding tag. Each hop contributes serialisation delay proportional to packet size and lane width, plus the receiver’s clock recovery and elastic buffer, plus the switch’s forwarding latency. The dominant terms for a small read are not bandwidth but hop count and the store-and-forward behaviour of each component; this is why a device behind a switch has higher latency than one attached directly, and why the topology matters as much as the lane count.

That topology is the subject of enumeration and bus-device-function addressing, where the same routing fields are used to discover what is attached. The rules that decide which transactions may pass which, and how a device orders its writes, are covered in posted writes and ordering rules. And the negotiation that decides whether a link runs at Gen3 x8 or Gen4 x4 in the first place is described in link training and negotiated width.

Failure modes worth knowing

A tag shortage looks like a performance cliff rather than an error: when a function has all its tags outstanding, further reads stall at the requester, and a driver that issues many small reads concurrently sees latency rise without any error indication. A credit shortage looks similar but is per class, so a flood of posted writes can starve completions only if the receiver’s buffers are shared in a way the implementation did not anticipate.

Malformed packets are handled at the receiver boundary. A packet with a bad CRC is dropped and NAKed; a packet with a valid CRC but an invalid header (an unknown type, an address outside any bridge’s window, a completion with no matching tag) is reported as a malformed or unexpected completion, and the requester may treat it as a fatal error. The distinction matters because the first is repaired by replay and the second is not.

Finally, the link width and speed are properties of the trained link, not of the slot. A card in a x16 slot that trains to x4 delivers a quarter of the expected bandwidth, and nothing in the software-visible configuration space necessarily reports the reason; the link status registers report the negotiated width and speed separately from the slot capability, and the two disagreeing is the first thing to check when throughput is lower than the slot suggests.