PCIe Topology and Bus-Device-Function Enumeration

Before any driver binds to a device, something has to discover that the device exists, assign it an address range, and record the path to it. On PCIe that work is done by walking a tree that the hardware itself defines, using configuration requests routed by a three-part identifier. The identifiers and the walk are inherited almost unchanged from PCI, which is why a modern device is still named by a bus number, a device number, and a function number even though no bus exists in the physical sense.

The 16-bit address that names a function

A PCIe function is addressed by a 16-bit identifier: eight bits of bus number, five bits of device number, and three bits of function number. The bus number is the routing key — a configuration request with a given bus number is forwarded by every bridge whose secondary-to-subordinate range contains it — while device and function select the target within that bus. Sixteen bits give 256 buses, 32 devices per bus, and 8 functions per device, for a maximum of 65,536 functions in one hierarchy.

The device number is not a physical slot in the sense that a person would use the word. On a root complex, each root port is a device on bus 0 with its own device number, so a machine with several root ports has several devices on bus 0 and nothing else. Behind a switch, each of the switch’s ports is a device on the switch’s upstream bus, and each downstream port opens a new bus for whatever is attached below it. A device number therefore identifies a position in the hierarchy, and the same device number appears at many different places in the tree.

Function number exists because one physical package may expose several independent functions. A three-bit field allows up to eight, and the common cases are a two-port network controller or a device that presents both a legacy and a modern interface. A function is the unit that owns configuration space, has its own base address registers, and gets its own driver; the device number is a packaging and routing convenience, not a software boundary. This is why a listing of PCI devices shows entries like 03:00.0 and 03:00.1 — two functions of one package, each with its own driver.

Configuration space and the two address mechanisms

Each function owns a 4 KiB configuration space. The first 256 bytes hold the legacy PCI header and its capability list, and the remaining 3 KiB is extended configuration space holding PCIe extended capabilities. Within the first 64 bytes sit the vendor and device identifiers, the class code, the six base address registers of a normal function, the interrupt line and pin, and the header type that says whether this is a normal function, a bridge, or a cardbus bridge. The class code is what lets software find a driver without knowing the vendor: a mass-storage controller is class 01, a network controller is class 02, and the subclass and programming interface refine those into something a driver can match on.

Reaching that space uses one of two mechanisms, and the choice matters for what the firmware can do before it knows the topology. The older mechanism uses two I/O ports: one to write an address selector containing the bus, device, function, and register offset, and one to read or write four bytes of data. Because it is an I/O-port protocol, it works before any memory mapping exists, which is why early firmware uses it. The newer mechanism maps configuration space into memory, using an ECAM region described by ACPI, and then a function’s configuration space is simply an offset from a base address: the bus number times a stride, plus device, function, and register. Memory-mapped access is faster and works for the full 4 KiB, but it requires the firmware to have already parsed the ACPI tables, so the two mechanisms coexist and the early boot path uses the port-based one.

The base address registers are where enumeration gets interesting. Each is 32 bits, and the low bits encode whether the region is memory or I/O and whether it is prefetchable. A device reports its required size by having software write all ones and read back: the device clears the bits it does not implement, so the resulting mask reveals the alignment and therefore the size.

/* Size a 32-bit memory BAR: write all ones, read the mask, restore. */
volatile uint32_t *bar = &cfg[bar_index];
uint32_t save = *bar;
*bar = 0xFFFFFFFFu;
uint32_t mask = *bar;
*bar = save;
if (mask == 0u || mask == 0xFFFFFFFFu)
        return 0;                        /* no BAR implemented here */
uint32_t size = ~(mask & ~0xFu) + 1u;    /* low four bits are flags */

A 32-bit memory BAR with a 16 MiB requirement reads back a mask with the low 24 bits clear. A 64-bit BAR consumes two consecutive BAR slots, which is why a device with large memory apertures has fewer usable BARs than the six the header allows. Nothing may be assigned until every BAR has been sized this way, because overlapping assignments are not detectable after the fact.

The walk

Enumeration is depth-first from bus 0. For each device number on the current bus, read the vendor identifier: a value of all ones means no device is present, which is how the walk detects empty positions without any sideband signal. For a present device, read every function. For each function, read the header type; a bridge header means the function has a secondary bus and the walk must descend, a normal header means it is a leaf.

Assigning bus numbers is the part that requires care, because a bridge’s registers define the range it forwards. A bridge has three: the primary bus it sits on, the secondary bus immediately below it, and a subordinate bus that bounds the highest bus number in the entire subtree below it. The walk assigns a fresh bus number to each bridge’s secondary bus as it descends, then on the way back up writes the largest number seen in that subtree into the subordinate register. Until that write happens, the bridge forwards only its secondary bus, so configuration requests to deeper buses do not reach them. The recursion order therefore matters: descend, enumerate the subtree, then program the range on the way out.

The result is a tree whose internal nodes are bridges and switches and whose leaves are endpoints. A root complex may have several root ports, each the root of an independent hierarchy with its own bus number range, which is why a machine can have multiple PCI domains — separate 16-bit identifier spaces that are not reachable from one another by configuration requests. A switch adds a level: its upstream port is a device on the bus above, and each downstream port opens a bus below, so a device behind two levels of switch is addressed by a bus number that only exists because enumeration created it.

What the OS does with the result

Firmware normally performs the walk and leaves a configured tree for the operating system, either through ACPI’s PCI root bridge objects or through a flattened device tree that describes the same ranges. The OS then re-walks to discover what is there. It normally leaves firmware assignments in place, because the firmware may have set up a framebuffer or a boot disk on those addresses and moving them would break the handoff, but it can reassign a bridge window when the firmware’s layout leaves a device unconfigured.

Address assignment itself is a bin-packing problem constrained by the apertures the host bridge exposes. A 64-bit prefetchable BAR must land in a prefetchable window above 4 GiB if one exists; a 32-bit BAR must land below 4 GiB; an I/O BAR must land in the I/O port window, which is typically small and is the resource that runs out first on a machine with many legacy devices. The kernel therefore sorts requests and assigns the constrained ones first, and a device whose BAR cannot be placed is left unconfigured — visible as a device with no driver bound rather than as an error at boot.

Failure modes and their signatures

The classic failure is a bridge whose subordinate bus register was never widened, so devices below it are invisible. The symptom is precise: the bridge is present and reports a secondary bus, but no function is found on any bus beyond it, and the range registers show a subordinate equal to the secondary. This happens when enumeration assigns bus numbers but does not complete the upward pass, and it is the reason the walk must be a proper recursion rather than a loop over bus numbers.

A second failure is a function that disappears between the vendor-ID read and the BAR sizing, which is possible with hot-plug or with a device that resets when its configuration space is accessed in an unexpected order. Reading the vendor ID again after sizing and treating a change as device removal is the defensive form.

A third is resource exhaustion that looks like a driver bug. When the 32-bit memory window or the I/O window is full, a new device cannot be assigned, and the kernel’s behaviour — skipping the device, or reassigning everything and disturbing the console — depends on policy. On machines with large BARs, such as a modern graphics card requesting a multi-gigabyte aperture, the 64-bit window must be large enough or the device falls back to a smaller aperture or fails to configure.

None of this requires a physical bus, but the naming survives because the software model does. A PCIe link’s negotiated width has no effect on the identifier: a x1 link to a switch port is still a bus, a device, and a function. The mechanics of that link are in PCIe lanes and packet layers, and the ordering rules that constrain how a configured device’s writes reach memory are in posted writes and ordering rules. The address translation that sits between a device’s DMA and system memory is covered in DMA addressing and the IOMMU.

Why the identifiers outlived the bus

A topology described by numbers rather than by physical wiring has one large advantage: it can be virtualised. A hypervisor can present a guest with a synthetic hierarchy whose bus numbers it chose, and a passthrough device can be placed at any position in that hierarchy as long as the guest’s view is internally consistent. This is why the same 16-bit identifier appears in ACPI tables, in device tree bindings, in hypervisor configuration, and in diagnostic tools decades after the electrical bus disappeared. The number is a name for a position in a tree, and a tree is what software needs.