Address Translation Across Host and Device

An address written by a kernel running on a GPU may be translated by up to three independent mechanisms before it reaches a memory cell: the GPU’s own memory management unit, the IOMMU’s second stage, and, on some systems, an interconnect’s system-level translation. Each stage exists for a different reason, and the arrangement of the stages explains most of the behavioural differences between unified-memory systems and passthrough systems.

The reason to care is that a translation stage is not free and not transparent. It has a granularity, a cache, an invalidation protocol, and a fault path, and each of those becomes visible in the presence of virtualization because the hypervisor owns one of the stages while the guest owns another.

Three stages and who owns each

The first stage is the GPU’s. A device driver builds page tables in ordinary memory and writes the root’s address into a device register. The device walks those tables itself, caching results in a translation cache. This stage exists so that a kernel can address buffers by virtual address and so that the driver can relocate buffers without rewriting command streams. In a virtual machine the addresses in these tables are guest-physical, because the guest driver does not know host-physical addresses.

The second stage is the IOMMU’s. It maps guest-physical addresses, or in the passthrough case device-supplied addresses, to host-physical frames. The hypervisor owns it entirely, and the guest cannot read or modify it. This is the stage that enforces the isolation invariant: whatever the guest puts in its own tables, the second stage decides which frames are reachable.

The third stage exists only on systems with a system-level interconnect that performs its own translation, such as a coherent fabric linking processors and accelerators. On such a system the device’s address may be resolved by the fabric rather than by an IOMMU, and the fabric’s translation is configured by the platform rather than by the hypervisor. Its presence changes the security argument, because the stage that enforces isolation is no longer the one the hypervisor controls.

Two of these stages can be collapsed. If the guest’s page tables already contain host-physical addresses, stage one and stage two coincide and only one walk happens. That is what happens when a guest is given a device whose driver is aware that it is virtualized and cooperates with the host to install real frames. The collapse removes a walk and removes the need to keep two tables consistent, and it requires the guest driver to be trusted with host-physical addresses, which is exactly what a passthrough guest is not.

Nested translation cost

When both stages are present, a single access walks two tables. If each table has four levels and each level’s entry misses in its cache, the access costs up to eight dependent memory reads before the data read begins. Devices mitigate this by caching intermediate translations as well as final ones, but the cache is smaller than a CPU’s and shared among all the work the device is doing.

The cost is not linear in the number of stages, because the caches absorb most accesses. The cost is concentrated in the working set that does not fit in the translation caches, which for a GPU is often the whole buffer: a kernel that streams through a multi-gigabyte array touches every page once and gains nothing from caching translations. That is the case where large pages matter most, and it is why both stages should use the largest page size the platform supports for large buffers. A 2 MiB page reduces the number of leaf entries by 512 compared with 4 KiB, and a 1 GiB entry reduces it by a further 512, which changes the walk count from one per 4 KiB to one per gigabyte.

Large pages constrain allocation, because the frames behind a large page must be physically contiguous or at least aligned and grouped. A hypervisor that backs guest memory with transparent huge pages gets this for free where the guest’s allocation pattern permits it; one that fragments guest memory does not, and the guest’s performance degrades for a reason that is invisible from inside the guest.

Faults as a communication channel

A translation miss can be handled in two ways. The hardware can fault, delivering the fault to software which maps the page and resumes the access, or the hardware can be told in advance that the page is not present and will never be, in which case the fault is a signal rather than an error.

The second form is what makes managed memory work. The device’s tables mark a page as invalid; the access raises a fault carrying the faulting address; the host driver learns the address, allocates a frame, migrates the contents, installs the translation, and resumes the faulting context. The fault is not an error path but the primary mechanism by which pages arrive. That inverts the usual assumption that faults are rare, and it means the fault path’s throughput is a first-order performance property of the system.

PCIe’s Page Request Interface exists to standardise the message a device sends when it needs a page. Without it, a device that faults has no way to tell the host which address it wanted except through a device-specific register interface, and a device-specific interface cannot be virtualized generically. With PRI, the request is a message on the bus carrying the requester’s ID and the address, and the host can attribute it to a tenant because the requester ID identifies the function. Address Translation Services is the companion: it lets a device request and cache translations with tags, and requires the host to invalidate by tag when a mapping changes.

Together, ATS and PRI let an accelerator behave like a CPU with respect to memory: it caches translations, faults on demand, and receives invalidation. The catch is that both mechanisms expand the set of things that must be correct. A stale ATS translation is a device writing to a frame that has been reallocated; a PRI request that arrives after the tenant has been destroyed is a request the host must refuse without crashing. Implementations handle this with a per-function invalidation protocol and with the IOMMU’s fault reporting queue, and these paths are exercised only under memory pressure, which makes them the ones that need the most careful testing.

What coherence changes

On a platform without hardware coherence between host and device, every transfer of a page requires cache maintenance: the host must clean its caches before the device reads the page, and the device’s writes must be flushed before the host reads. This is the reason a migration is not merely a copy but a sequence of copy plus maintenance operations, and it is why non-coherent systems have higher per-page migration cost.

On a coherent platform the maintenance disappears, and with it the requirement that pages be migrated at all. A device can access host memory directly with the same semantics as a CPU, so “device memory” becomes a placement decision rather than an architectural distinction. The performance difference between placing a page in device memory and in host memory becomes a function of the interconnect’s bandwidth and latency, and the software can move pages for performance reasons without correctness concerns.

The virtualization consequence is significant: a coherent device’s addresses can be resolved by the same page tables the CPU uses, so the two-stage model collapses to one in the common case, and the hypervisor’s second stage does the only translation. That removes an entire class of bug, at the price of requiring the platform’s fabric to be trustworthy, since the fabric is now the enforcement point.

Migration and the translation state

Migrating a guest that holds device memory requires capturing not just the memory contents but the translation state that makes those contents reachable. The device’s page tables must be saved and re-established at the destination, and the frames they name must be at the same addresses or the tables must be rewritten.

This is where the three-stage arrangement becomes an obstacle. The guest’s tables contain guest-physical addresses, which survive migration because the guest’s physical address space is what is being preserved. The IOMMU’s second stage does not survive, because it names host frames that are allocated at the destination. The hypervisor can therefore migrate the guest’s tables unchanged and rebuild its own, which is a clean separation and the reason guest-physical addressing is worth the extra walk.

A device that has cached ATS translations must have them invalidated before the migration, or the destination will receive a device still holding translations to the source’s frames. The invalidation is part of the stop-and-copy phase, and its cost is proportional to the number of cached translations, which is why devices with large translation caches have a longer migration stop. The oversubscription machinery relies on the same invalidation path, and a system that gets invalidation right for migration usually gets it right for eviction as well. The state-capture sequence for a migrating guest depends on this invalidation being acknowledged rather than merely issued.