Link Training and the Width a PCIe Link Settles On

Before any transaction layer packet crosses a PCIe link, the two ends have to agree on how many lanes are usable, which lane is which, which direction each is running, and at what rate symbols will be sent. None of this is negotiated by a management processor or written by software; the two physical layers do it themselves with a state machine that runs on every link from the moment power is applied. The result is a link that comes up at some width and speed that may be lower than either end’s capability. Software can read the result, and it can ask for a retrain or a different target rate, but it cannot force a width that the lanes will not support.

What the state machine does on the way to L0

The link training and status state machine begins in Detect, where each end senses whether a receiver is present by looking for the far end’s termination. When both sides detect a partner, they enter Polling: each transmits a fixed pattern of ordered sets while trying to receive and align to the far end’s ordered sets. Polling is where bit and symbol alignment are established, using the encoding’s alignment characters so that the receiver knows where symbol boundaries are.

Configuration follows. Here the two ends exchange ordered sets that carry each side’s link number and lane number, so that the physical lanes can be mapped onto logical lane numbers. Two things are resolved in this state. Width is decided by what both ends can support and what lanes are actually functioning: each side advertises its maximum width, and the link comes up at the largest common value that has a working lane set. Lane ordering is resolved because board routing may connect a device’s lane 0 to the slot’s lane 3; the exchange lets the two ends reverse or permute the lanes so that logical lane 0 is logical lane 0 on both sides.

Once Configuration completes, the link enters L0, the normal operating state, and packets can flow. The states on either side of that are worth knowing because error signatures point at them. Recovery is a re-training state entered whenever the link must resynchronise — after enough CRC errors that the replay buffer is exhausted, after a speed change, after a lane goes down. Disabled and Hot Reset are the states for a link brought down deliberately; L1 and L2 are low-power states where the transmitter is electrically idle and re-entry to L0 takes a training sequence rather than a single handshake.

Where width is chosen, and how it degrades

The negotiated width is not the slot’s width and not the card’s width; it is the smaller of the two capabilities, further reduced by whatever lanes fail to train. A card designed for eight lanes in a sixteen-lane slot comes up at eight, which is expected and costs nothing. A card designed for sixteen lanes that comes up at four has a problem: the ordered sets in Configuration only count lanes that completed the exchange, so a lane with a broken connection, a shorted differential pair, or a marginal signal is simply excluded from the trained set.

That behaviour is deliberate and it is also dangerous. A link that trains to a subset of its lanes still works, so a damaged connection produces a working device at reduced bandwidth rather than a failure. The evidence is in the link status registers, which report the negotiated width and speed separately from the link capability registers. When the two disagree, the cause is on the physical side, and the common ones are a poorly seated card, a riser or extender that adds loss, an adapter that wires fewer lanes than the slot provides, and a lane whose AC coupling capacitors or connectors are damaged.

Degradation also happens in the other direction: a slot can be electrically split so that its sixteen lanes form two independent eight-lane links, which is how a platform presents more devices than it has x16 slots. This bifurcation is a board-level property, chosen by strapping or firmware, and it is invisible from the card’s point of view — the card sees eight lanes and reports eight lanes.

Rate changes and the equalization handshake

Speed is negotiated with the same mechanism as width but with a stricter condition: both ends must request the higher rate. An ordered set carries a speed-change bit, and the link only moves to a higher rate when both sides have set it. A card that supports a higher generation than its partner runs at the partner’s rate, which is why an older adapter upstream in the path caps the whole link.

Increasing the rate is not a matter of changing a clock divider. At 8 GT/s and above, the signal at the receiver is no longer clean enough to sample with a fixed threshold, because the channel attenuates high frequencies and the resulting inter-symbol interference smears each symbol into its neighbours. The specification therefore requires an equalization procedure as part of the move to the higher rate. Each transmitter applies pre-emphasis or de-emphasis — deliberately boosting the first bit of a transition and attenuating the bits that follow — while each receiver applies continuous-time filtering and, at higher rates, decision feedback equalisation, in which the receiver subtracts the influence of previously decided symbols from the current sample.

The procedure is a handshake over the same ordered sets: one side proposes transmitter settings, the other measures and reports, and the two iterate until the far end reports an acceptable result, or until the attempt fails and the link falls back to the previous rate. The tuning takes longer at higher rates, because the channel loses more of its margin, equalisation has more to recover, and there are more presets to try. This is why a link can come up at the higher width but the lower speed: the lane set is fine, the rate change is not.

A rate change is not instantaneous from software’s view either. Moving to the new rate requires re-entering Recovery and retraining at the new clock, during which no packets cross the link. A device doing this periodically — for power or for thermal reasons — pays that interruption, which is one reason rate changes are not made casually.

Retimers, redrivers, and the length of the path

A link that is electrically long, such as one routed across a board, through a cable, and into a backplane, may need a component in the middle. A redriver is an analogue amplifier: it reshapes the signal without understanding it, so the link still trains end to end and the state machine is unaware of it. A retimer is a full physical layer: it trains its own link to each side, recovers the clock, decodes the ordered sets, and retransmits. A retimer therefore appears to the link state machine as an additional participant, and the specification provides for it explicitly, including the means by which each end learns how many retimers are in the path. Retimers add latency, because each one buffers and retransmits, and they add a point of failure, because the training must succeed on both sub-links.

The practical consequence for diagnostics is that a failure in the middle of a path is consistent with a healthy endpoint. A device that never reaches L0 in a riser or a cable assembly but trains normally when plugged directly into the slot is a statement about the path, not the device, and the equalization procedure is usually where it fails: the extra loss consumes the margin the transmitter settings need.

Reading the negotiated result

Software sees the outcome, not the negotiation. The link capability register reports what the component supports, the link status register reports the negotiated width and speed, and the link control register lets software request a rate change or trigger a retrain. Comparing capability against status is the first diagnostic step for any bandwidth complaint, and it distinguishes three situations cleanly: the link negotiated exactly what both ends support, in which case the limit is the device or the slot; the link is narrower than the smaller capability, which is a physical problem; or the link is at a lower rate than both support, which is an equalization failure or an upstream component that caps the rate.

Two further details matter for the relay of that state to the operating system. First, a hot-plug event does not change the connection until the link is retrained, so the presence detect and the link state can disagree momentarily, and a driver that reacts to presence detect alone may see a device before the link is ready. Second, a link that has been brought down for power has to go through training again on wake, and the negotiated width after the wake may be lower than before, because a lane that was marginal when cold may fail after a few thermal cycles. Nothing in the protocol remembers the previous trained width; each training run starts from capability.

The upstream mechanisms that assume a trained link are the ordering rules of posted writes and the packet layers of PCIe lanes and packet layers. Where the trained link sits in the hierarchy is decided by enumeration and bus-device-function addressing. The state machine is not the only self-configuring mechanism on a link; the encoding and framing changes in later generations, described in flit mode and PAM4 encoding, alter what the physical layer must recover before the state machine can even complete.

When training fails outright

Failure to reach L0 at all leaves the link in Detect or Polling, and the link status register reports that training has not completed rather than a specific cause: the Link Training bit stays set while the state machine is in Configuration or Recovery. The causes are systematic rather than marginal: a lane whose polarity is inverted without the mechanism that would compensate for it, a reference clock that is absent on one side, two endpoints that both believe they are downstream ports of a switch, or a device held in reset by a signal the platform did not deassert. Polarity inversion is normally handled automatically, which is why a reversed pair usually trains and only a lane-order problem that exceeds the permitted permutation fails.

The last failure mode is the most confusing to diagnose because it looks like a software problem. A switch whose downstream port has never trained is a bridge with no secondary link, so the devices behind it do not exist as far as enumeration is concerned, and the symptom is a missing device rather than a malfunctioning one. There is no bus error because there was never a bus.

Training also depends on the reference clock being within a tight tolerance relative to the partner’s, which is why a spread-spectrum setting mismatch between two ends prevents a link from coming up while both ends individually test as functional. The state machine cannot report this; it reports only that it failed, and the distinguishing evidence is that the failure is reproducible between two specific components and not with either of them separately.