Non-Transparent Bridges and Multi-Host PCIe Topologies

A PCIe hierarchy is a tree with exactly one configuration master at the root. Every address a device sees has been assigned by that master, every bus number exists because that master created it, and the walk described in PCIe topology and BDF enumeration assumes there is nobody else assigning addresses. Connect two such trees together and the assumption fails immediately: both roots enumerate, both assign overlapping ranges, and neither can be the other’s device. Making the connection work requires a component that hides one tree from the other while still moving data between them.

Why a cable between two hosts does not work

If the link between two root complexes is a plain link, each side sees an endpoint where it expects a tree, and a root port expects its downstream partner to be a device or a switch. An endpoint that is itself a root complex will not respond to configuration requests in the way a switch does: it has no secondary bus registers to describe a subtree it knows nothing about, and it has been programmed by a different master with a different address map.

Even if the two sides agreed on bus numbers, the address problem remains. Each root assigns memory windows to its own devices without knowledge of the other’s assignments, so a device below root A may be assigned the same memory address range as a device below root B. A forwarded transaction carries only an address, so it would be ambiguous which side’s device was the target, and a window that overlaps means a write intended for one host’s device lands in the other’s.

The last obstacle is that both roots are masters. A configuration request must be claimed by exactly one target, and a completion must return to the requester; with two trees that each believe they own the fabric, requester identification, error reporting, and interrupt routing all become ambiguous. The fix is to make the connection look like a device to each side while remaining a bridge in function — which is what a non-transparent bridge does.

What each side sees

A non-transparent bridge presents two faces. On each side it appears as a bridge function: a type 1 header with primary, secondary, and subordinate bus registers, so that the local root enumerates it as a bridge with a small subtree below. That subtree is deliberately minimal (the bridge itself and, depending on the implementation, a small set of registers) so that the local master believes it has found an ordinary bridge and stops there rather than trying to walk into the other host’s tree.

Behind that face the bridge translates addresses in both directions. Each direction has a set of memory windows: a local address range that the local master may access, and a translated base address that is used on the far side. A write from host A to an address inside a window is forwarded to host B with the address rebased into host B’s address space. Host B’s own addresses are chosen by host B’s enumeration, so the two sides can be completely independent, and the window is the only overlap between them.

The windows are typically large — the granularity is set by the matching bits of the translated base register, so a window is at least an alignment unit, and implementations support windows of megabytes or more; they are usually configured out of band by the platform firmware or by one side as part of a handshake, because neither side can discover the other’s address map by configuration space alone. The bridge may also expose an inbound translation window from the far side, so each host can be given a window into the other’s memory. Each direction is configured independently, which is what makes the address maps independent.

Doorbells, scratchpads, and the handshake

Moving bytes is not sufficient; the two sides need to signal each other. A second set of registers provides one-way and peer-doorbell registers: a write to a doorbell register on one side is translated by the bridge into an interrupt delivered to the other side’s local interrupt controller. Implementations commonly provide a self-doorbell as well, so the writer can confirm that its own doorbell write actually went out, which is needed because the write is posted and has no completion.

A third set of registers, the scratchpad, is ordinary read/write storage inside the bridge, visible from both sides. It is the medium for the handshake that must happen before any data moves: one side writes a value, the other reads it, and the pair establish which windows are active, what the peer’s address map is, and whether the peer is alive. Because the scratchpad is visible from both sides, it also serves as a heartbeat — a sequence number incremented periodically by each side and checked by the other to detect a peer that has stopped without the link going down.

The ordering rules of posted writes apply in full across the bridge. A data write destined for the peer’s memory and a doorbell that announces it are both posted transactions, and within one traffic class the doorbell cannot pass the data write on the link. The gap is at the bridge: the doorbell is consumed by the bridge and turned into an interrupt, while the data write is forwarded to the far side, so the interrupt can be raised before the forwarded write has been committed at its destination. The software protocol therefore validates a sequence number in the data structure rather than trusting the interrupt alone. A read-back between the two helps in the requester’s direction, because a read cannot pass an earlier posted write, but it proves only that the write was accepted, not that it landed on the far side.

/* Publish a descriptor, order it with a read-back, then ring the doorbell. */
desc->len  = len;
desc->addr = dma_addr;
wmb();                                  /* keep the descriptor stores in order */
(void)readl(&ntb->scratchpad[0]);       /* a read cannot pass the earlier writes */
writel(1u, &ntb->doorbell);             /* bridge may raise the interrupt first */

This is the same hazard as a device signalling completion to its driver, made worse by the extra hop.

No coherence, and what that forbids

A non-transparent bridge moves memory transactions; it does not make the two hosts coherent. Nothing in the bridge keeps host A’s caches consistent with writes performed by host B through a window, so a line that host A has cached will not be invalidated when host B writes that memory. The consequence is that a shared data structure across the bridge cannot be manipulated with ordinary loads and stores plus a lock; the locking would have to be coherent, and it is not.

The usable programming model is therefore message-passing with explicit publication. A producer writes a buffer, issues a read-back or a fence, sets a sequence number in a structure that is not cached by the consumer, and rings the doorbell. The consumer invalidates or avoids caching the publication structure, reads the sequence number, and only then reads the data. Caching must be managed explicitly: the window region is usually mapped as uncached or write-combining on both sides so that reads and writes go to the bridge rather than to a cache, at a real cost in the throughput of small accesses to that region. Large transfers are copied into the window with streaming writes, which use the write-combining path and are efficient, while the handshake structures stay in a small uncached region.

This also means the bridge provides no atomicity across the two address spaces. An atomic read-modify-write that the local fabric offers within one hierarchy does not exist across the bridge, because the operation would have to be executed by the far side under its own rules. Every cross-bridge synchronisation has to be built from messages, and every protocol that assumes a compare-and-swap succeeds in doing so only within one host’s memory.

What happens when one side stops

The characteristic failure is asymmetric death. If the link between the two sides goes down, each side sees a link-down event on the port that hosts the bridge, and the memory windows that pointed across become invalid. Writes posted into those windows before the link went down are lost, because posted writes have no completion and a link that is down cannot report what it did not deliver. A protocol built on the doorbell being seen after the data will hang rather than corrupt, which is why the heartbeat is a sequence number and not a flag: a flag that is already set tells the reader nothing about how many updates it missed.

A second failure is the stale window. If the surviving side reassigns the window base to a local purpose — for instance, after re-enumerating the subtree — while the far side still believes the window is live, transactions arrive at physical addresses that are now something else. This is the same class of error as a stale IOMMU translation, and the defence is the same: the protocol must establish liveness before using a window and must tear the window down from the side that owns the address space.

A third is the split brain. Two hosts that share a peripheral through an NTB can both decide they own it, particularly after a link flap that makes each side believe the other is gone. The bridge does not arbitrate ownership; it has no notion of which side is primary. Ownership therefore has to be carried in the protocol, with a lease and a tie-breaker, and a design that relies on the link state to determine ownership will fail during exactly the event it was meant to survive.

A fourth is address aliasing between the window and a device. Because the local master assigns the window as part of its own address map, a window placed into a region that also contains a device aperture produces two responders for one address, and the resulting behaviour depends on ordering rather than on policy. Correct placement is a platform responsibility, and the failure is visible as intermittent data from the wrong target rather than as a clean error.

Where this sits among the alternatives

Three other approaches connect hosts or share devices, and the trade between them is the useful comparison. A network link between hosts carries messages with the same programming model and a much larger latency and per-message cost, but requires no shared address map. Two functions of one device, split so that each host owns one, shares a peripheral without sharing an address space, which is the mechanism described in SR-IOV and function partitioning, and it works only when the device supports partitioning. A coherent interconnect such as CXL removes the incoherence that makes the non-transparent bridge hard to program, at the cost of requiring both sides to implement a cache coherence protocol and of accepting the latency of a coherent access to remote memory.

The non-transparent bridge is what remains when the requirement is to move data between two independent hosts, each of which insists on being the configuration master, on hardware that already exists. Its cost is that everything above the physical layer (synchronisation, ownership, discovery, teardown) becomes the software’s problem, and the hardware’s contribution is a pair of address windows and a doorbell.