SR-IOV Virtual Functions and Static Partitioning of GPU Engines

Passthrough gives one guest one card. To give several guests slices of one card without a hypervisor in the data path, the hardware itself must present several PCI functions, each with its own configuration space, its own register window, its own interrupt vectors, and its own address space. SR-IOV is the standard shape for that, and it is the design that combines direct hardware access with an interface the guest operating system can enumerate as an ordinary device.

The specification’s central move is to split one physical function into two roles. The Physical Function is a complete PCI function: it owns the device, has a full configuration space, and is driven by the host. Virtual Functions are lightweight functions derived from it. Each has a Routing ID of its own, so each can be addressed and assigned independently, but each has only the small slice of configuration space that the specification defines, and none of them can be configured except through the Physical Function.

What a virtual function actually owns

The SR-IOV Extended Capability lives in the Physical Function’s configuration space and contains the numbers that define the split: the total number of virtual functions, the number currently enabled, the stride between their Routing IDs, and the base address of the Virtual Function BARs. Enabling a virtual function is a write to a bit in that capability. Nothing else about the function’s identity is programmable from the virtual function itself.

The BAR arrangement is the part that constrains everything above it. Virtual functions do not have independent BAR registers in the usual sense; their BARs are described in an array inside the Physical Function’s capability, and the array’s total size is the size of one BAR multiplied by the number of virtual functions. A GPU that gives each virtual function a 4 MiB register window and supports 16 of them needs a 64 MiB region in the Physical Function’s BAR space before a single virtual function is enabled. This is why SR-IOV on large-BAR devices consumes address space quickly, and why the number of virtual functions a card can expose is often limited by aperture rather than by silicon.

Configuration space access is deliberately asymmetric. A virtual function’s configuration space carries a Type 0 header with its own vendor and device ID, its own command register, and its own MSI-X capability. It carries no SR-IOV capability, no power management capability, and no writable BAR registers, because its BARs are described in the Physical Function’s capability rather than by the function itself. Everything that would let a tenant reconfigure the device is absent by construction, which is the point: the virtual function is a function whose policy surface has been removed rather than a function that is politely asked not to use it.

Routing IDs and the eight-function wall

A PCI Requester ID is a 16-bit value: 8 bits of bus, 5 bits of device, 3 bits of function. Three bits gives eight functions per device, which is fewer than SR-IOV wants. The Alternative Routing-ID Interpretation capability widens the function field to 8 bits, permitting 256 functions on a bus and freeing the device field to be used as part of the function number. ARI is effectively mandatory for a GPU exposing more than seven virtual functions, and it changes how the host enumerates the bus: a hierarchy operating in ARI mode walks functions in a single pass rather than probing eight possible functions per device.

The Routing ID matters beyond enumeration because it is the key for two other mechanisms. The IOMMU’s context table is indexed by it, so each virtual function gets its own second-stage translation table and therefore its own isolated address space. The interrupt remapping table is bound to it as well: a request is translated into a host vector only when the requester identifier it carries matches the entry, so a virtual function’s MSI-X vectors cannot be forged by another tenant. Both properties come free once the function has a distinct Routing ID, and neither is available to a design that multiplexes many tenants through one function.

What the GPU must replicate

The specification says nothing about what a device must implement behind each virtual function. For a GPU the question is which pieces of state carry tenant identity, because each of those pieces must exist once per function.

The first is the translation context. A virtual function needs its own page table root, its own set of translation caches, and the ability to have those caches invalidated without disturbing another tenant’s. Hardware that shares a single translation cache across the card can still partition it by tagging entries with a context identifier, but the tag must be present in the cache and the invalidation must be able to select by tag. Without that, one tenant’s unmap flushes everyone’s translations, and the cost of a mapping change becomes proportional to the number of tenants.

The second is the command processor. Each virtual function needs its own queue of submissions and its own doorbell. The doorbell must be a distinct address, or a distinct offset in a shared page, so that a write by one tenant cannot be interpreted as work for another. A GPU whose command processor has a single global queue cannot be partitioned this way at all; the queue itself must be replicated in hardware.

The third is the memory partition. A virtual function’s view of device memory must be disjoint, or must at least be a distinct address window that the hardware checks. If two tenants can name the same physical device memory address, isolation has failed regardless of what the PCI layer says, because a compute kernel’s operands are addresses the guest chose.

The fourth, and the one that limits granularity hardest, is everything the card does not have more than one of. Display engines, video decode blocks, copy engines, and the memory controllers are typically shared. A card with one display engine cannot give two tenants independent display output through SR-IOV; one of them gets the engine and the other gets a compute-only slice. This asymmetry is why virtual functions on GPUs are usually not interchangeable: some carry graphics capability, some do not, and the host must know which is which before assigning them.

The partitioning is static, and that is the design

Because each virtual function owns fixed hardware resources, the split is decided when the virtual functions are enabled and does not change while they are assigned. A card divided into four equal slices gives each tenant a quarter of the execution units and a quarter of device memory, and a tenant with a small workload leaves three quarters of its slice idle while its neighbour saturates.

This is not an oversight. Dynamic repartitioning requires the ability to move execution units and memory between tenants at runtime, which requires preemption of running work and the ability to relocate device memory contents, which requires either a unified address space the hardware can remap or a copy through the host. Static partitioning avoids all of it: no preemption, no migration, no scheduler. In exchange it accepts fragmentation, and the fragmentation is visible as a utilisation loss that grows with the variance in tenant demand.

The other cost is that assignment is not revocable without a reset. Reclaiming a virtual function from a guest requires the guest to stop using it, the function to be reset, and its memory to be cleared before another tenant can be given it, because otherwise the next tenant inherits the previous one’s data. This is the same reset problem that afflicts plain passthrough, amplified by the fact that a single card may hold several tenants’ worth of stale state at once.

Where virtual functions stop being the right answer

A virtual function is a good fit when tenants are long-lived, their resource needs are known, and the operator is willing to pay for a fixed slice to get direct hardware access. It is a poor fit when tenants are short-lived or when demand is bursty, because the slice cannot follow the demand and the assignment cannot be moved.

The gap between “one card, one guest” and “one card, four fixed slices” is filled by two other designs. Mediated devices put a host-side scheduler in front of a shared function, giving up direct access in exchange for dynamic allocation and migratability. Hardware slicing that partitions the execution and memory resources into isolated instances within a single function, described under fixed-slice partitioning, sits between the two: isolation without separate PCI functions, and therefore without the ability to hand a slice to a guest as a device.

The decision between them is a decision about which of three properties to sacrifice: direct access, dynamic allocation, or isolation strength. SR-IOV sacrifices dynamic allocation and keeps the other two. That is a narrow but very useful corner, and it is why the mechanism survives despite the address-space cost and the static split.