The Interception Surface of a Virtual GPU

A network device is virtualizable because its interface is narrow: a ring of descriptors in memory, a doorbell write announcing new work, and an interrupt announcing completion. A GPU presents the same three categories of interface, and every one of them is far larger. The register file behind its base address register is not a page of control bits but a region large enough to hold thousands of distinct registers, several of which have read side effects. The DMA engine does not fetch fixed-size descriptors; it fetches command buffers encoding a complete instruction set with its own opcodes, predicates, and memory operands. The interrupt source is not one line but a table of vectors, one per channel or per queue.

Virtualizing a GPU therefore reduces to one design question asked three times: for each of those interfaces, does the guest talk to hardware directly, or does something in the host stand in the path? Passthrough answers “hardware” three times, full emulation answers “host” three times, and every other design mixes the answers.

The register file behind the BAR

A PCI function advertises its memory-mapped register file through base address registers. BAR sizes are powers of two and are discovered by writing all ones and reading back the mask, so a function that needs 256 MiB of register space must be granted an aligned 256 MiB window. A 32-bit BAR cannot describe more than 2 GiB, which is why large GPUs use 64-bit BARs and why the resizable-BAR capability exists: on some access paths the aperture that must be mapped is the whole of device memory.

The registers inside that window are not equivalent. Most are ordinary read/write state. A smaller set are doorbells: a write of a queue index or an offset that tells the device new work is available. A doorbell write is posted; there is no completion and no read-back, so the driver cannot tell whether the device observed it. That property makes doorbells cheap on real hardware and expensive in a virtual machine, because the only way to see one is to trap it.

Every access to a mapped BAR from a guest is either permitted by hardware or trapped. On x86 a second-level translation fault, or an access to a page the hypervisor withheld, causes a VM exit costing hundreds to a few thousand cycles, against a handful of cycles for the same store executed natively. Emulating a submission path register by register therefore costs more than the work it submits. Designs that emulate the register file survive only because they move bulk submission out of the register file and into shared memory, which is exactly the structure virtio uses for its GPU device.

The mapping itself is unremarkable in userspace:

#include <fcntl.h>
#include <stdint.h>
#include <sys/mman.h>
#include <unistd.h>

int ring_doorbell(uint32_t queue_index)
{
    /* resource0 is the function's first BAR, exposed by the PCI core */
    int fd = open("/sys/bus/pci/devices/0000:3b:00.0/resource0", O_RDWR | O_SYNC);
    if (fd < 0)
        return -1;
    /* two pages mapped, so the doorbell at 0x1000 lies inside the window */
    void *bar = mmap(NULL, 8192, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
    if (bar == MAP_FAILED)
        return -1;
    /* the mapping keeps the BAR alive after the descriptor is closed */
    close(fd);
    /* posted write: no completion, no error reported to this process */
    *(volatile uint32_t *)((char *)bar + 0x1000) = queue_index;
    return 0;
}

The same sysfs resource file is what a hypervisor maps into a guest when it grants a passthrough window, with the physical frames behind it pinned and owned by that guest.

The address the device computes

The second interface is memory. A modern GPU has its own MMU and walks page tables that the driver builds in ordinary memory. A submission names buffers by GPU virtual address; the device translates those addresses itself, using a page table root the driver installed by writing a register.

Inside a virtual machine the addresses the guest driver writes into those page tables are guest-physical, not host-physical. Two translations therefore sit in series: the GPU’s own walk, then the IOMMU’s second stage. Nothing requires the first stage to be validated by the hypervisor, and that is the danger. A page table is data the guest writes and hardware consumes without interpretation. If the second stage is an identity mapping, translating guest-physical addresses to the same host-physical addresses, then whatever frame the guest names in a leaf entry is the frame the device writes, and the guest has DMA to host memory.

The classical answer is shadowing: write-protect the pages holding guest page tables, trap every modification, and maintain a parallel table whose entries hold host-physical addresses. For a CPU this is tolerable because page tables change rarely. For a GPU it is not, because buffer mapping happens continuously and every map is a page table write. The alternative, and the one widely deployed passthrough designs use, is to leave guest page tables alone and let the IOMMU’s second stage supply the safety property. Guest addresses stay guest-physical; the IOMMU translates them into whatever frames the host allocated. The hypervisor’s job moves from validating tables to owning the second stage, which is a much smaller and more auditable piece of work.

The second stage is not free either. Its page tables are themselves in memory, they consume cache and TLB entries, and a GPU workload that touches a large working set will miss in the IOMMU’s translation cache as well as its own. Implementations mitigate this with device-side translation caches and with larger mapping granularity, which is the same trade a CPU makes between huge pages and page-table walk cost.

Interrupts and the vector table

A GPU signals completion with MSI-X, a posted memory write to an address the host nominated, carrying a small data payload that selects a vector. There is no shared line to arbitrate, which is why MSI-X scales; the table size field in the capability is eleven bits, so one function can name up to 2048 distinct vectors.

Sharing one physical function between tenants breaks the one-vector-one-owner assumption. Either the hardware tags a vector with the context that raised it, in which case the host routes by tag, or the host traps the interrupt and demultiplexes by reading device state, which adds a round trip to every completion. Legacy INTx signalling is worse: the line is level-triggered and shared, so the host must mask it, let the guest’s handler run, and re-assert on end-of-interrupt. Getting that sequence wrong yields either lost interrupts or an interrupt storm that consumes the machine.

MSI is also a DMA write, and this is the part that surprises people. A device that can write any address can forge an interrupt for any vector, including one belonging to another tenant or to the hypervisor. The IOMMU’s interrupt remapping table closes that hole: the vector the guest selected is translated through a per-device table into a host vector, and a write outside the table is dropped. Interrupt remapping is not an optimisation. Without it, passthrough is a privilege escalation primitive.

Config space and the driver that must match

The fourth interface is the easiest to overlook: the 256-byte configuration space, extended to 4 KiB under ECAM. It is where the function identifies itself with a vendor and device ID, where BARs are sized, where the MSI-X capability lives, and where reset and power management capabilities sit. A guest operating system enumerates it and binds a driver by matching those identifiers.

Emulated devices must therefore lie consistently. The virtio GPU presents a vendor and device ID that the guest’s virtio driver claims, and describes BARs the emulator backs. A passthrough function presents its real identifiers and the guest’s real vendor driver binds, which is why passthrough requires the guest to have that driver and why the host cannot pass through a function whose driver it also wants to use.

Capabilities are a second problem. A guest that reads the SR-IOV extended capability can attempt to change the number of virtual functions; a guest that reads the power management capability can attempt to change device power state. The host must either hide those capabilities or intercept writes to them, and hiding a capability is not free, because some drivers refuse to operate without one they expect. The rule that emerges is that emulated config space must be a consistent fiction: complete enough to bind the driver, restrictive enough that nothing the driver can write changes state the host owns.

Why the interfaces cannot be split carelessly

The interfaces look separable and are not, because the device’s trust boundary is defined by what it can do to memory, not by which registers it exposes. A hypervisor that emulates the doorbell but passes through the queue memory has protected nothing: the guest writes a command buffer whose operands point wherever it likes, and the device executes it. A hypervisor that passes through the BAR but traps DMA has protected everything and gained nothing, because the device faults on its first descriptor fetch.

The invariant that holds across all four interfaces is that the set of addresses the device may touch must equal the set the guest was granted, and it must hold for page table walks, command buffer fetches, data transfers, and interrupt writes alike. Emulation enforces it by never letting the device see guest-controlled addresses at all. Passthrough enforces it in the IOMMU. Partitioned designs enforce it by giving each tenant a function whose address space is already disjoint from every other.

That invariant also determines the cost structure. The passthrough design pushes the whole problem into the IOMMU and pays in lost sharing. The SR-IOV design pushes it into per-function isolation hardware and pays in fixed partitioning. The mediated design keeps scheduling-relevant registers in the host and pays in traps on the submission path. None of them avoids the invariant; they differ only in which layer is trusted to enforce it.