SR-IOV and How One Device Becomes Many Functions

A PCIe function is the unit of ownership. It has one configuration space, one set of base address registers, one driver bound to it, and one requester ID that names it on the link. That is a clean model until a device needs to serve several independent consumers (several virtual machines, several containers, or a host and a management processor), at which point one function per device means one owner per device, and everyone else gets an emulated copy. Single root I/O virtualisation changes the arithmetic by letting one physical device instantiate many lightweight functions, each with its own configuration space and its own identity on the link.

The physical function as a controller

The device’s real function is called the physical function, and it keeps everything a normal function has: its BARs, its full configuration space, its driver. What it gains is a capability structure describing a set of virtual functions it can create. That structure contains the number of virtual functions currently enabled, the stride between their configuration spaces, an offset used to compute their bus and device numbers, the device identifier they report, and, critically, the base address register requirements for a virtual function.

The physical function’s driver is the controller. It decides how many virtual functions to enable, it can disable them all, and it is responsible for the device-level configuration that all of them share — which physical port a virtual function belongs to, how bandwidth is divided, which hardware resources are partitioned among them. A virtual function has no ability to change the number of virtual functions or to alter the capability that created it; its own configuration space does not contain the capability, because it is a product of it rather than a manager of it.

That asymmetry is the whole design. A virtual function is deliberately minimal: a small configuration space, a few base address registers, its own message-signalled interrupt table, and its share of the device’s queues. It is cheap enough to instantiate hundreds of times, which is what makes it usable as a per-virtual-machine device, and it is restricted enough that a misbehaving virtual function cannot reconfigure the physical device out from under its siblings.

Where a virtual function’s configuration space lives

A virtual function’s configuration space is not a separate 4 KiB allocation in the sense that the physical function’s is; the platform computes the address from parameters in the physical function’s capability. The routing identifier of the first virtual function is the physical function’s identifier plus the offset, and each subsequent one is the previous plus the stride, with the addition carrying into the bus number when the device and function fields overflow. That is why enabling many virtual functions can consume device numbers on buses adjacent to the physical function’s.

The memory cost is the part that surprises people. A base address register belongs to a function, so N virtual functions need N copies of every virtual function base address register, and the physical function’s capability describes the requirement for one virtual function, which the platform multiplies by N. A device that wants 128 virtual functions each with a few megabytes of register aperture is asking the platform for hundreds of megabytes of address space, and the platform must find it as one contiguous, naturally aligned region because the virtual functions are laid out at a fixed stride. This is the reason high virtual function counts appear mostly on devices whose virtual functions need small windows, and the reason implementations often offer a choice between many virtual functions with small apertures and fewer with large ones.

Enabling them is a write to the number-of-virtual-functions field followed by a write to the enable bit, and the platform must then re-run address assignment for the newly created functions. Some platforms require a rescan for the new functions to appear at all, because enumeration happens at boot and the new functions did not exist when the tree was walked. A device whose virtual functions appear only after a rescan is normal; a device that never exposes them usually has the capability disabled by firmware, since a virtual function is also a DMA requester and must be covered by the translation hardware described in DMA addressing and the IOMMU.

ARI and the eight-function ceiling

The identifier that names a function is eight bits of bus, five bits of device, and three bits of function. Three bits allow eight functions per device, which was sized for the era when a device might have two ports and a management interface. A network adapter with sixty-four virtual functions does not fit.

Alternative Routing-ID Interpretation removes the ceiling by reinterpreting the eight bits of device and function as a single eight-bit function number, giving 256 functions on one bus at one device number, addressed as bus:function pairs. The device reports the capability, the upstream component has to agree to forward in the reinterpreted form, and a switch that does not support it will treat the higher function numbers as empty devices. This is why virtual functions above the seventh can be invisible through an old switch or a riser that does not implement the forwarding rule, and the symptom is precise: functions below eight work and the rest are absent, with no error.

The eight-bit function number interacts with routing in a way worth stating plainly. Routing by bus number still works as before, but routing by requester ID inside the device becomes a lookup on eight bits rather than five plus three, so the hardware that steers an incoming completion back to the right function must be built for the wider field. A device that implements the capability has to implement that wider steering in its own transaction handling, because completions for a virtual function arrive at the link and must be delivered to the right one.

Isolation in the fabric, sharing in the device

Every virtual function has its own requester ID, and therefore its own translation context in the IOMMU. That gives each virtual function an address space that no other can reach: a driver that programs a bad address produces a translation fault rather than a write into another consumer’s memory. The requirement for that isolation to hold is that the translation and interrupt-remapping structures be sized for the number of functions and that the platform disable any bypass path for the affected device. A function that is accidentally left in identity-mapped mode — a policy often used for trusted devices — has no isolation at all, and the fact that it is a virtual function does not change that.

Functional isolation is not resource isolation. Virtual functions typically share the physical port, the packet buffers, the scheduler, and the link. A virtual function that transmits at line rate consumes the link that the others need; a device with a shared buffer pool can be driven into a state where one function’s memory pressure affects the others. The specification’s purpose is to give each consumer a clean configuration and control interface and a protected address space, not to promise separate bandwidth or separate queues. Which aspects are genuinely partitioned is a device property, and the device’s documentation, not the capability structure, is where it is stated.

There is also a shared failure domain. A physical function reset, a firmware update, or a link retrain takes down every virtual function on the device at once, and no amount of per-function reset changes that. Function-level reset, which both physical and virtual functions are required to support, resets one function without disturbing the others, but it cannot recover from a device-wide fault, and a design that treats a virtual function as an independent device will be surprised by correlated failures.

Operating pitfalls

The first pitfall is changing the virtual function count while functions are in use. The number-of-virtual-functions field is writable by the physical function’s driver, and reducing it while a virtual function has a driver bound destroys a live device. The interface expects the count to change only when the virtual functions are idle, and there is no mechanism in the hardware to enforce that.

The second is interrupt vector exhaustion. Each virtual function has its own message-signalled interrupt table, and the platform’s interrupt remapping structures must have room for every vector across every function. A device with large vector counts per virtual function and many virtual functions can exhaust the platform’s remapping capacity, and the failure appears as a virtual function that cannot allocate interrupts rather than as a device error.

The third is address space defragmentation. Because virtual function apertures are laid out contiguously at a fixed stride, the platform must have a contiguous region large enough for all of them. A machine that has already assigned most of its 64-bit aperture to a graphics card and a large storage controller may be unable to fit a hundred virtual functions, and the failure is at enable time, after the number was set — so the driver must be prepared for the enable to fail and fall back to a smaller count.

The fourth is the interaction with hot-plug and migration. A virtual function can be reset and reassigned, and some devices support migrating a virtual function’s state between physical devices, but the interface for that is optional and its presence is a capability. Treating a virtual function as migratable because it is virtual is a mistake; the capability has to be present and a manager has to implement the protocol.

Where it fits

Three mechanisms now divide a device among consumers, and the choice is about where the boundary is drawn. A non-transparent bridge, described in multi-host topologies, divides a device between whole hosts that each insist on being a configuration master. A shared virtual address space over a coherent interconnect removes the copy by letting a device and a process share addresses and page tables. SR-IOV sits between them: the device is divided in the fabric, at the function granularity, with a hardware identity per slice, and no coherence between slices beyond what the device itself provides. It asks little of the rest of the system, and it works with existing root complexes, existing switches, and operating systems whose enumeration already understands functions.