Paravirtual GPUs and the Mediated Device Lifecycle

SR-IOV splits a device by replicating the state that carries identity. That works when the hardware has enough replicas to go around, and it fails when the thing being shared is a single execution engine, a single memory controller, or a single copy engine. The alternative is to keep the device whole and interpose software between the guest and the hardware, so that the host decides which guest’s work runs and when. A device virtualized this way is a mediated device: it looks like a PCI function to the guest, it is assigned through the same interface as a passthrough function, and every access it makes is translated or trapped by a driver in the host.

The structure that makes this workable is a split between a control path and a data path. The control path carries configuration, allocation, and scheduling decisions and is allowed to be slow, because it runs at the granularity of a context creation or a guest boot. The data path carries submissions and completions and must not be slow, because it runs at the granularity of individual commands. Every mediated design is an argument about where to put the boundary between the two, and the failure modes follow from where that boundary lands.

The object model the kernel defines

A mediated device is not a PCI function discovered by enumeration. It is created by writing a UUID into the create attribute of one of the types the parent driver advertises, under that parent’s directory in sysfs. Each type names a set of attributes that describe what the instance will get, and the instance is created only when those attributes have been satisfied. The result is a device that can then be bound to the VFIO driver and opened by a virtual machine monitor, which maps its BARs into the guest and receives its interrupts.

The creation path is plain file I/O, which is why it can be scripted and why failures surface as write errors rather than as device status:

parent=/sys/class/mdev_bus/0000:3b:00.0
type=$parent/mdev_supported_types/$(ls $parent/mdev_supported_types | head -n 1)

# the types this parent offers, and how many instances of the chosen one remain
ls $parent/mdev_supported_types
cat $type/available_instances

# request one instance; the UUID becomes a directory under the mdev bus
echo 0b1f8a2e-6c43-4d1e-9a5f-2c7d9e0a1b34 > $type/create

# destroying it later is a write of 1 to the instance's remove attribute
echo 1 > /sys/bus/mdev/devices/0b1f8a2e-6c43-4d1e-9a5f-2c7d9e0a1b34/remove

available_instances is the admission control: the write to create fails when the parent’s capacity for that type is exhausted, and the count is recomputed as instances are created and destroyed. The count is not a promise about performance, only about whether the parent can allocate the resources the type names.

The three-part structure of parent, type, and instance exists because the parent driver is the only code that knows what a valid configuration is. The generic layer can create a UUID and hand out a file descriptor; it cannot know how much device memory a particular type should have, or whether the parent has capacity left. So the type publishes attributes, the creation path validates them, and the parent allocates real resources. This inverts the usual device model: instead of the hardware announcing what it is, the software asks for a shape and the driver either grants it or refuses.

Teardown is the mirror image and is where the interesting guarantees live. Destroying a mediated device must revoke every mapping it holds before the instance is freed, must stop any work it has queued, and must clear whatever state in the parent was allocated to it. If teardown is incomplete, a subsequent instance can inherit a mapping to frames now owned by another guest, or find that the parent’s scheduler still has a context it believes is live. Because the destroy path is invoked when a guest exits, including on a guest crash, it must be idempotent and must not depend on the guest cooperating.

Where the data path runs

The design question that determines performance is whether the guest writes its submissions into a region of device memory that the host has mapped for it, or into host memory that the host then copies. The first is faster and requires the hardware to accept work from a guest-supplied address; the second is slower and is the only option when the hardware insists on addresses in its own memory space.

In the copy case, each submission is read out of the guest’s memory by the host, validated, and placed in a queue the device reads. The cost is one copy per submission plus the trap that announces it. For a workload that submits a few thousand large kernels, this is negligible. For a workload dominated by many small operations, as a graphics workload is when each draw is encoded as its own submission, the per-operation cost dominates and the design collapses.

In the direct case, the guest’s queue lives in a region the host has pinned and mapped into both address spaces, and the doorbell write is trapped. The trap is still per submission, but it is one exit rather than a copy plus an exit. Designs that need to reduce it further use a shared page where the guest writes a queue tail index and the host polls, with a notification only when the queue transitions from empty to non-empty. That turns the per-submission cost into a per-batch cost, at the price of latency when the batch is short.

Neither case lets the guest name addresses the device will dereference directly unless those addresses were translated. If the guest’s command buffer contains a GPU virtual address, the host must either rewrite it or arrange for the device’s own translation to resolve it. Rewriting requires parsing the command stream, which requires knowing the instruction encoding, which is precisely the vendor-private knowledge that a mediated driver has and a generic one does not. This is why mediated GPU drivers are device-specific even though the framework around them is generic.

Interrupts through a shared function

A mediated device on a shared physical function has no MSI-X vectors of its own in the sense that a virtual function does. The function has a vector table, and the driver must partition it among instances or use a shared vector with demultiplexing. Partitioning is preferable: instance A’s completions arrive on vector 3, instance B’s on vector 4, and no host intervention is needed at interrupt time. It requires the hardware to let software assign vectors per context, which most modern GPUs do because they already need it for their own queue model.

Demultiplexing is the fallback. The host receives the shared interrupt, reads device state to determine which contexts have completed work, and signals the corresponding guests. Each interrupt then costs at least one register read, which on a device behind a bridge is a round trip measured in hundreds of nanoseconds, and the read is serialising. Under a workload that completes thousands of small operations per second, this becomes the bottleneck rather than the submission path.

The interrupt must also be attributed correctly for the guest’s own bookkeeping. A guest that receives an interrupt it did not cause will run a handler for a queue that has no work, which is harmless but wasteful; a guest that does not receive an interrupt it did cause will stall until a timeout. Both failure modes appear when vector assignment races with instance creation, which is why the assignment must complete before the instance is exposed to the guest.

What the guest sees, and what it cannot see

The guest sees a PCI function with a vendor and device ID chosen by the mediated driver, a BAR layout the driver describes, and a set of capabilities it wants the guest driver to find. It does not see the physical GPU’s real identifiers unless the design deliberately mirrors them, and it does not see the physical function’s capabilities for SR-IOV, power management, or resets.

This fiction has to be complete enough for a real driver to bind. A vendor driver checks identifiers, reads a capability chain, maps BARs, and then programs the device assuming it is alone. If the emulated register file is missing a register the driver reads at probe time, the driver either fails or, worse, reads a default value and proceeds with a wrong assumption. The practical consequence is that mediated GPU drivers are written by the vendor, because only the vendor knows which registers a probe sequence touches.

The guest also cannot see the other tenants, which is the isolation property. It cannot enumerate them, cannot read their memory, and cannot influence their scheduling except by consuming shared capacity. The last of those is not a security boundary and should not be treated as one: a tenant that saturates the shared engine delays its neighbours, and no mediated design prevents that without an explicit arbitration policy, which is a separate problem from isolation.

Why the model persists

The mediated model exists because it is the only one that supports three properties at once: dynamic allocation of a shared device, migration of a guest that holds it, and a host that retains visibility. Dynamic allocation comes from the fact that instances are created and destroyed at runtime and share the parent’s resources. Migration comes from the fact that the instance’s state can be serialised by the driver, because the driver owns all of it. Visibility comes from the fact that the host sees every submission, which is what makes accounting and scheduling possible.

The price is that the data path is never as fast as direct access, and the price is paid per operation rather than per byte. That is why mediated designs are a good fit for workloads with coarse-grained submissions and a poor fit for graphics, where the driver’s submission rate is high and latency-sensitive. The fixed-slice partitioning approach recovers the direct path by making the hardware enforce isolation instead of the host, and the passthrough approach recovers it by giving up sharing entirely.