Shared-Bus Arbitration and the Clock Ceiling That Ended It

A bus is a set of wires that several devices drive, and that single fact determines almost everything else about it. Because the wires are shared, only one device may drive them at a time. Because the wires are parallel, all of them must be valid at the receiver within one clock period. Because many of them are bidirectional, every change of direction costs dead cycles. The peripheral bus that personal computers used through the 1990s, PCI, is the clearest worked example of the design, and its replacement by point-to-point serial links is a case of a shared medium meeting a physical wall rather than a logical one.

Address phase, turnaround, data phase

PCI multiplexes address and data on the same 32 wires, AD[31:0]. A transaction begins when a master asserts FRAME# and places the target address on AD together with the command encoding. Every other device samples those wires on the next rising clock edge, so every transaction is broadcast: there is no destination field, no routing decision, and no way for a device to remain unaware of traffic except by not responding to it. The intended target claims the transaction by asserting DEVSEL#, and the data phases follow. Each data beat completes only when the initiator has asserted IRDY# and the target has asserted TRDY# at the same edge, which is how wait states are inserted without any sideband negotiation: either side simply withholds its ready signal.

A read costs an extra turnaround cycle, because the wires that carried the address must switch direction before the target can drive data back. A write does not pay that cost, since address and data travel the same way. This asymmetry is visible in the arithmetic. At 33 MHz with 32 data bits, one clock moves four bytes, giving a theoretical 132 MB/s, conventionally quoted as 133. A single 32-bit read with one wait state occupies an address phase, a turnaround, and a data phase, so four clocks move four bytes — a quarter of peak. Burst transfers amortise the address phase across many beats, which is why every bus of this generation rewards long bursts and why a device that cannot burst is expensive to keep fed.

The arbiter and the grant

Only one master may drive at a time, so the bus needs a policy for choosing. PCI uses a central arbiter, usually inside the host bridge, with a dedicated pair of wires per master: REQ# to ask, GNT# to grant. Arbitration for the next transaction happens during the address phase of the current one, so the grant is already asserted when the bus goes idle. That overlap is what keeps the bus from losing a cycle to every decision, and it means the arbiter must know the shape of the transaction currently in flight — a master that requests a long burst delays every other grant by that burst.

The specification deliberately leaves the policy unspecified. Round-robin gives every master a bounded wait; fixed priority gives the latency-critical device the shortest wait and starves the rest. Neither is wrong, because the bus is a shared resource and worst-case latency is the sum of every other master’s bursts plus arbitration overhead. A design that needs a hard upper bound on latency has to impose a maximum burst length or a token rotation, and both cost throughput on the transfers that could have run longer. That tension is the reason a graphics card and a disk controller on the same bus make each other slower in a way that is not visible in either device’s own timing.

The arbiter also has to handle the master that is granted the bus and then does nothing. If a master asserts GNT# and fails to start a transaction within a fixed number of clocks, the arbiter may park it (drive its grant continuously while it is idle), so that a later request does not pay an arbitration round trip. Parking is a throughput optimisation with a fairness hazard: a parked master that suddenly becomes busy already owns the bus, and everyone else waits one more transaction.

Why the wires, not the protocol, set the speed

Raising the clock is the obvious way to raise bandwidth, and it is where the shared bus runs out of room. In a synchronous parallel bus, thirty-two data wires plus control and parity must all be stable at every receiver inside one clock period. A signal in ordinary FR4 board material travels roughly 15 cm per nanosecond, so at 33 MHz a 30 ns period tolerates a trace path of several metres; at 66 MHz the period is 15 ns; at 133 MHz it is 7.5 ns, and the flight time across a loaded backplane plus connector plus input buffer is comparable to the entire period.

Three effects close the budget from different directions. First, each device added to the bus adds input capacitance (a few picofarads per pin, tens of picofarads across a connector), which slows every edge and raises the current the driver must source. Second, every device sits on a stub, and every stub is an impedance discontinuity that reflects part of the wave back toward the driver, so the signal does not settle monotonically; it rings, and the receiver must wait for the ringing to fall inside its threshold window. Third, skew between wires accumulates: the same clock edge arrives at different receivers at different times, and the data wires do not all have identical lengths, so the effective window in which all bits are simultaneously valid shrinks faster than the period does.

The specification’s timing budgets therefore come with load limits. A 33 MHz segment is characterised for something on the order of ten electrical loads, with fewer permitted as the clock rises and the trace budget tightens; 64-bit segments cut the allowance again because there are twice as many wires to skew-align. The practical ceiling for the parallel multiplexed bus landed near 533 MB/s for 64-bit at 66 MHz, and PCI-X reached about 1 GB/s at 133 MHz by adding an attribute phase and split transactions and by restricting each segment to a small number of loads. Each step bought a factor of two and cost another slice of the load budget.

What point-to-point changed, and what it cost

The replacement inverts the topology. Instead of one set of wires shared by every device, each link connects exactly one transmitter to one receiver. That removes arbitration entirely, removes the multi-drop stub and its reflections, and removes the parallel skew problem because the clock is recovered from the data stream rather than distributed across thirty-two wires. Differential pairs carry each bit as a voltage difference between two wires, so common-mode noise coupled into both is subtracted at the receiver. Encoding adds redundancy: 8b/10b in the first two generations maps each byte to a 10-bit symbol with bounded running disparity, which keeps the DC balance sane and guarantees enough transitions for the receiver’s clock recovery; 128b/130b in later generations cuts that overhead by sharing a two-bit sync header across 128 bits of payload.

The trade is visible in three places. Pin count per device falls — a four-lane link is eight wires plus grounds, against roughly fifty mandatory signals for 32-bit PCI including arbitration and interrupts — but the switch that fans those links out is now a required component, and it adds a hop of latency and its own power. Per-transaction latency rises for small accesses, because the request must be serialised, packetised, transmitted, deserialised, and routed before a device sees it; a single-register read that took a few bus clocks now takes hundreds of nanoseconds. And bandwidth becomes per-link rather than shared: a x1 link at 2.5 GT/s with 8b/10b encoding carries 250 MB/s in each direction, about the same order as the 133 MB/s of 32-bit PCI but dedicated to one device rather than arbitrated, and a device with four lanes gets a full gigabyte per second to its switch port, with contention pushed to the uplink instead of to every transaction.

What breaks on a shared bus

The failure modes follow from broadcast and from shared ownership. If no device asserts DEVSEL#, the master terminates the transaction as a master abort and, on a read, returns all ones; a driver that does not check the status sees 0xFFFFFFFF as data and may treat it as a valid value. Parity is computed across address and data phases and reported through PERR# for data and SERR# for address and system errors, but parity detects and does not correct: a transaction with a parity error is not retried by the hardware, and recovery is left to software or to a higher-level protocol.

Locked transactions are the structural hazard. A master can assert LOCK# to hold a target for an atomic read-modify-write sequence, and while the lock is held every other master is blocked from that target. Two masters that each lock a different target and then request the other’s deadlock, and no arbitration policy can resolve it because neither will release. Bridges make this worse in a way that is easy to miss: a transaction that crosses a PCI-to-PCI bridge occupies the bus on both sides of the bridge, so bandwidth below a bridge is not additional to bandwidth above it — it is nested inside it, and a hierarchy of bridges multiplies the occupancy of a single upstream transfer.

Live insertion exposes the last weakness. Adding a card to a running bus means its pins touch a set of wires that other devices are actively driving, and the resulting glitch can corrupt a transaction in flight on a completely unrelated device. Hot-plug on a shared bus requires staggered pin lengths so ground and power connect before signal, plus separate power control per slot — a workaround for a medium that has no concept of a point-to-point connection. Point-to-point links make the same problem tractable, because a newly inserted device disturbs only its own link, which the training sequence described in link training and negotiated width can bring up from scratch. How those links carry transactions is the subject of PCIe lanes and packet layers, and how a host discovers what is attached is covered in enumeration and bus-device-function addressing.