PCI Passthrough and the Boundaries an IOMMU Actually Draws

Passthrough is the virtualization design with the least software in it: the guest is given a PCI function, a window of its register file, and permission to receive its interrupts, and then the guest’s own driver programs the hardware exactly as it would on bare metal. Everything the guest submits reaches the device without interpretation. That is the appeal, and it is also the reason the design is only safe when something outside the device can bound the addresses it may touch.

On a machine without an IOMMU, a bus-mastering device can write to any physical address. Granting one to a guest therefore grants the guest write access to the hypervisor’s memory, to other guests’ memory, and to the firmware. The historical workaround was to trust the guest, which is not a virtualization design but an absence of one. Every serious passthrough implementation depends on an IOMMU, and the interesting engineering is in deciding what a single IOMMU domain can safely contain.

The device does not arrive alone

A PCIe topology is a tree of switches and bridges, and DMA is not confined to the subtree a device sits in. Two functions behind the same switch can address each other directly through peer-to-peer transactions, which do not traverse the root complex and are therefore not translated by an IOMMU that only covers upstream traffic. A function that can initiate a peer-to-peer write to a sibling’s BAR can corrupt that sibling’s state, including state the sibling’s driver believes is protected.

PCIe Access Control Services exist to make this explicit. The ACS capability on a switch or root port advertises which redirections it can block: peer-to-peer requests between downstream ports, completions routed upstream, and so on. Where ACS is implemented and enabled, the hierarchy behaves as a set of isolated subtrees. Where it is not, every function behind the same bridge shares a trust domain whether or not the software wishes it.

The kernel’s IOMMU group is exactly this trust domain materialised as an object. A group is the smallest set of devices that the IOMMU can isolate from all other sets, computed from the topology and the ACS capabilities of the bridges above each device. A device may only be assigned to a guest if every other device in its group is assigned to the same guest, or is unbound from any driver and therefore not issuing DMA. /sys/kernel/iommu_groups/ exposes the grouping, and the practical consequence is familiar: a GPU and the audio function that shares its root port land in one group, so passing the GPU through means passing the audio function through as well, or accepting that neither can be assigned.

The grouping is visible from userspace, which is how a binding decision is made before the function is offered to a guest:

# each group directory lists the devices that cannot be separated from each other
for g in /sys/kernel/iommu_groups/*/; do
    printf '%s: ' "$(basename $g)"
    ls "$g/devices/"
done

There is a tempting way around this, which is to declare that the bridges do not implement ACS and let the software pretend isolation exists. It produces working configurations and unsound ones, because the peer-to-peer path it ignores is a real path. The grouping is a report about hardware, not a policy that software gets to override.

What the second stage translates

Intel’s VT-d and AMD’s IOMMU both provide two-stage translation for devices: a first stage that a guest may control for its own device-visible addresses, and a second stage that only the hypervisor controls. In passthrough configurations the first stage is typically bypassed and the second stage does the work, mapping guest-physical addresses to the host frames backing the guest’s memory. The device walks those tables itself, using a root structure the hypervisor installs and a context entry per function that names the second-stage table root and the domain identifier.

Translation granularity is 4 KiB at the leaves, with large-page entries at 2 MiB and 1 GiB in the usual x86 arrangement. Large pages matter more for a GPU than for a network card, because a device that walks its own tables on every buffer access pays the walk cost in the middle of a kernel. A compute kernel touching a multi-gigabyte buffer through 4 KiB mappings generates an enormous number of table walks, and the walk itself consumes memory bandwidth that the kernel wanted. Backing a guest’s memory with huge pages, and mapping it into the IOMMU at the same granularity, removes most of that overhead.

The tables are not static. When a guest’s memory is ballooned, migrated, or when a page is swapped, the host must remove the mapping before reusing the frame. If it does not, the device retains a valid translation to a frame now owned by someone else, and the guest’s driver will happily write through it. This is the passthrough analogue of a use-after-free, and it is why the unmap path is required to invalidate the device’s cached translation, not merely the IOMMU’s.

Caches that outlive the unmap

A device that walks page tables will cache the results, and the cache is not coherent with the IOMMU’s tables. Removing a mapping requires an invalidation that reaches the device, either by invalidating the IOMMU’s own translation cache and relying on hardware to propagate, or by an explicit device-side invalidation. VT-d uses a queued invalidation interface: the software writes descriptors into a ring in memory and the IOMMU consumes them in order, which allows batching but makes the completion a separate event to wait on.

Address Translation Services push this further by letting the device hold translations and revalidate them itself. A device with ATS may request a translation, cache it with a tag, and use it without an IOMMU lookup on each access; the IOMMU is then required to invalidate by tag when the mapping changes. The Page Request Interface is the companion mechanism for the opposite direction: instead of faulting silently on an unmapped address, a device with PRI can ask the host to map the page and retry. Both features exist because devices with large working sets want to behave like CPUs, and both widen the correctness surface, because a stale cached translation is indistinguishable from a valid one until it corrupts something.

For passthrough of a GPU the practical guidance is conservative: ATS and PRI are used where the platform and the device both support them correctly, and disabled otherwise. The cost of disabling them is performance, not correctness.

Resetting a function before reuse

Assigning a function to a guest requires that it start from a known state. Otherwise the previous owner’s command queues remain in device memory, its page tables remain installed, and its interrupts remain enabled for a guest that no longer exists. PCIe defines a Function Level Reset capability, which resets the function’s state without resetting the link or the rest of the hierarchy.

GPUs are notoriously poor at this. A function-level reset may leave memory controllers, display engines, or firmware state untouched, so the device returns with stale contents that the next driver does not expect. Implementations deal with this in three ways: refusing to reassign a function that lacks a usable reset, performing a bus reset of the whole slot and accepting the collateral damage to siblings, or invoking device-specific reset logic through a vendor-provided interface. The last is the reason passthrough support is device-specific even when the platform support is generic.

The related hazard is that a reset does not necessarily stop in-flight DMA. A device with queued writes may continue to issue them while the host tears down its IOMMU domain, so the teardown must either wait for the device to quiesce or ensure that translations are revoked before the frames are freed.

What passthrough gives up

The trade is stark and worth naming precisely. Passthrough provides the guest with the device’s full performance, the vendor’s complete driver, and every feature the hardware has, including ones the hypervisor does not know about. It provides no sharing: the function belongs to one guest for the duration, and the host loses the ability to use it, to inspect its memory, or to migrate the guest while the device is assigned. Live migration of a passthrough guest is not possible without either a shadow device or a mechanism for capturing and replaying device state, which no general solution provides.

The consequence is that passthrough suits workloads where a GPU is dedicated to a long-lived guest and where the operational cost of losing it is acceptable. It is a poor fit for a platform that wants to oversubscribe a pool of GPUs across short-lived tenants, which is why the SR-IOV partitioning model exists and why the mediated device model keeps a host-side scheduler in the loop.

The interface contract described in the interception surface of a virtual GPU still holds here, with the IOMMU as its sole enforcer. Passthrough is not the absence of virtualization; it is the relocation of the enforcement point from software the hypervisor controls to hardware the platform controls.