Flit Mode and PAM4: How PCIe Reached 64 GT/s

Every previous PCIe generation increased the link rate by making the transmitter switch faster. From 2.5 to 5 and then 8 and then 16 and 32 gigatransfers per second, the signalling stayed binary — two voltage levels, one bit per symbol per lane — and the rate rose. That approach ran out of headroom at 32 GT/s, not because transmitters cannot switch faster, but because the channel attenuates the high frequencies a faster binary signal needs. The next step changed the modulation instead, and that change ripples all the way up through framing and retry.

Why the symbol rate could not keep doubling

A binary signal at rate R has its energy concentrated around R/2 — the Nyquist frequency the channel must pass for the eye to be open. Copper traces, vias, connectors, and cables attenuate high frequencies more than low ones: the loss per unit length rises roughly with the square root of frequency for the skin-effect part and linearly with frequency for the dielectric part. Doubling the rate moves the Nyquist frequency up by a factor of two, and on a path already marginal at the lower rate, that is several more decibels of loss to recover with equalisation.

Equalisation cannot recover what is not there. A receiver can boost high frequencies but it also boosts the noise that shares that band, and the decision-feedback equaliser that subtracts post-cursor interference works on decisions that must already be mostly correct. At some rate, the combination of loss, crosstalk, reflections, and jitter leaves an eye that is too small to sample reliably with two levels. The specification’s answer was to stop raising the symbol rate for a doubling of throughput and to carry more bits per symbol instead.

Four levels, half the symbol rate

Pulse amplitude modulation with four levels encodes two bits in each symbol. To reach 64 GT/s, the link runs at 32 gigasymbols per second, and the Nyquist content of that signal sits at 16 GHz — the same frequency that a 32 GT/s binary signal occupies. The channel therefore sees the same loss profile it already handled at the previous generation, while the delivered bit rate doubles. This is the whole reason PAM4 exists in the standard: the electrical problem is held constant and the information per symbol is increased.

The price is paid in margin. Four evenly spaced levels give each eye a third of the peak-to-peak amplitude of a binary signal, and signal-to-noise ratio falls by a factor of three — about 9.5 dB. Every impairment that was previously a minor eye closure now consumes budget that does not exist, which is why the higher-rate links are specified with tighter jitter limits, better connectors, and shorter permitted channels. The receiver also needs a much better analog front end: a linear equaliser to open the eye and a decision-feedback equaliser with enough taps to remove the interference from several preceding symbols, all of which must adapt continuously as temperature and voltage drift.

Correction has to come before detection

Earlier generations handled errors after the fact. A packet that failed its link CRC was reported by sequence number and replayed by the transmitter from a buffer. That works when errors are rare: a link specified for a bit error rate on the order of one in a trillion loses so few packets that replay costs almost nothing in bandwidth, and the retry machinery is idle.

At PAM4 SNR levels the raw error rate before correction would be orders of magnitude worse, and replay alone would be ruinous, because a single error costs the retransmission of a whole frame plus the round trip to detect it. The specification therefore places forward error correction in the physical layer: an error-correcting code over the transmitted block that allows the receiver to repair a limited number of symbol errors without asking for anything. The code is interleaved so that a burst of errors caused by a single noise event is spread across several codewords rather than corrupting one codeword beyond repair.

The layering changes what an error means. A corrected error is invisible to the data link layer: the packet arrives intact and no replay happens. An uncorrectable error is detected rather than repaired, and only then does the replay machinery act. Links therefore carry corrected-error counters that rise long before the link fails, and a rising count is the earliest warning that a path is losing margin; the link still works, and will keep working until the uncorrected rate exceeds what replay can absorb, at which point the link retrains.

Replay at this layer also has a coarse granularity, because the retransmission unit is no longer a variable-length packet. In flit mode the transmitter’s replay buffer holds flits, and an uncorrectable error in one flit means every packet inside that flit is sent again. A flit packs many small transactions, so an error re-sends traffic that was itself undamaged. The error-correction layer absorbs most of this, and the design intent is that uncorrectable errors remain rare enough that the coarse retry unit does not matter.

Fixed-size flits instead of framed packets

The framing change is as significant as the modulation change. In previous generations, each transaction layer packet was framed on the wire with its own start and end tokens and carried its own CRC, and idle symbols filled the gaps when there was nothing to send. Packets of wildly different sizes meant variable serialisation delay and an idle-filling problem at high rates, where the overhead of framing becomes a measurable fraction of the link.

In flit mode the link transmits fixed-size flits. A flit is a fixed 256-byte unit that contains whatever transaction layer packets and data link layer messages are pending, packed together without individual framing, plus a CRC covering the flit. Idle content is represented inside the flit rather than between flits, so the link never needs to insert or remove fill symbols, and the serialisation delay of any given packet is bounded by the flit size rather than by the queue behind it. The data link layer’s acknowledgement and replay operate on flits, which is why the replay buffer is measured in them.

A fixed unit also makes the error correction tractable. The code operates over a known-size block with a known relationship to the CRC, so the receiver can run correction and verification as a stage whose latency is fixed. That is what allows the physical layer to correct without the transaction layer ever seeing an error, and it is the reason the two changes, PAM4 and flits, arrived together rather than independently: the correction scheme needs a fixed block, and the fixed block needs a modulation whose error rate justifies correction.

What changes at the endpoints

The highest rate is available only in flit mode, and flit mode is negotiated between the two ends in the same way rate and width are: both ends must support it, and a link that cannot agree runs at the highest rate both support in the older framing. A device that supports the previous generation’s top rate is therefore not limited to half the new rate; it runs at its own rate with its own framing, and the two ends simply agree on the lower common configuration.

For the device designer the change is concentrated in the physical layer and the data link layer: the transmitter needs a linear driver with four levels and programmable equalisation, the receiver needs a linear front end, an adaptive decision-feedback equaliser, and a decoder for the correction code, and the replay buffer is counted in flits rather than packets. The transaction layer is largely untouched — the header formats, routing, and ordering rules keep working, because the flit is a container. A packet’s behaviour is unchanged; only its journey is.

The latency accounting changes in ways that matter for some devices. A fixed flit means a small read’s request may wait for the current flit to be built rather than being transmitted immediately, and the error correction adds a fixed decode delay before the flit’s contents are released to the data link layer. Both are small compared with the round trip of a completion, but they are not zero, and a component that compares generations on bandwidth alone will find the newer link slightly worse for single small accesses even while it is far better for streams.

The equalisation and correction machinery is the same family of problems that link training solves at lower rates, and the framing assumptions about what is inside a flit are described from the other direction in PCIe lanes and packet layers. The link-level reliability that flit replay provides is still hop-by-hop: end-to-end integrity, where it is wanted, remains the responsibility of the optional end-to-end CRC in the transaction layer.

The efficiency ledger

It is worth keeping the overheads of each generation in one place, because the headline rates are misleading. A first-generation lane at 2.5 GT/s spends a fifth of its bandwidth on 8b/10b encoding and carries 250 MB/s. A second-generation lane at 5 GT/s carries 500 MB/s before payload. A third-generation lane at 8 GT/s with 128b/130b encoding carries roughly 985 MB/s, because the encoding tax fell to about 1.5 percent. A fifth-generation lane at 32 GT/s carries approximately 4 GB/s. A sixth-generation lane at 64 GT/s carries about 8 GB/s in raw terms, less the error-correction parity, the flit CRC, and the header and link-layer overheads that every generation pays.

The comparison to make is not raw rate but usable payload per pin and per watt, and there the story is one of diminishing returns: each doubling of rate costs more equalisation, tighter channel requirements, more error-correction machinery, and more power in the serialiser and the receiver front end. The trade the standard makes is that the same physical channel and the same connector can carry roughly thirty times the usable bandwidth of the first generation, at the cost of a link that no longer works without correction and no longer trains to the same margin on a marginal path.