Live Migration of a Guest That Holds a GPU

Migrating a guest without an accelerator is a solved problem: stop the guest, copy its memory, copy its CPU state, restart it elsewhere. The iterative pre-copy refinement reduces the stop by copying memory while the guest runs and re-copying only the pages that changed, converging when the dirty set is small enough to transfer in one bounded pass. An accelerator breaks both halves of that: it has state that is not in guest memory and not visible to the hypervisor, and it writes to guest memory continuously while the migration is in progress.

The designs that permit migration at all are the ones where the host can observe and control the device. A passthrough function cannot be migrated, because its state lives in registers and memory the host cannot read, and because the guest’s driver holds pointers to device memory addresses that do not exist at the destination. A mediated device can be migrated, because its driver owns every piece of state that matters and can serialise it. A fixed partition can be migrated only if the destination card has an identically shaped free instance, which makes migration a placement constraint rather than a data movement problem.

The four state domains

The state of a GPU-attached guest partitions into four domains, and each has a different capture mechanism.

The first is guest memory: the pages the guest’s driver allocated for command buffers, descriptor rings, and data. These are ordinary guest pages, so the hypervisor’s existing dirty-tracking mechanism covers them. The complication is volume and rate: a GPU writing at full bandwidth dirties guest pages far faster than a CPU workload does, so the dirty set during pre-copy may not shrink between passes.

The second is device memory: the pages the tenant holds in the accelerator’s own memory. These are not guest pages and are not tracked by the hypervisor. Capturing them requires the device driver to enumerate the allocations it made on behalf of this tenant and to read them out. For a mediated device this is a driver operation; for a passthrough device it is impossible.

The third is the device’s architectural state: the contents of the registers that define the tenant’s execution context, the page table root, the queue base addresses and indices, and the interrupt configuration. This state is small, typically a few kilobytes, and it is the state that must be restored before the device can be handed to the guest at the destination.

The fourth is in-flight work: kernels that have been submitted and not completed. This is the domain with no clean solution. A kernel’s intermediate state lives in registers and shared memory across thousands of threads, and capturing it requires either hardware support for context save or a guarantee that the kernel will be complete before the capture.

Ordering the capture

The order is forced by dependencies. Capturing guest memory before stopping the device produces a copy that the device may modify while it is being copied. Stopping the device before capturing its memory requires a mechanism to stop it, which requires the driver, which requires the guest to cooperate.

The sequence that works:

1. Quiesce the device. The host driver stops issuing new work for this tenant and waits for submitted work to complete, or, where preemption exists, preempts it. The completion must be verified rather than assumed, because a queued command may still be executing.
2. Invalidate the device’s translations for the tenant’s memory. This must be acknowledged by the device, since a cached translation would let a straggling access reach a page after the page has been copied.
3. Capture the device’s architectural state and the contents of its memory allocations. At this point nothing on the device can change either.
4. Capture guest memory. Because the device is quiesced and its translations are invalidated, guest memory is now stable except for CPU activity, which the guest’s own stop handles.
5. Transfer, then install in the reverse order at the destination.

Steps 1 through 3 must be inside the stop-and-copy window, because they cannot be done while the guest runs. Step 4 can overlap with pre-copy for the pages that were already copied, but the final pass must be inside the window as well. The minimum stop is therefore the time to quiesce plus the time to read the device’s memory plus the time to transfer the dirty guest pages.

Making the stop small

The size of the stop is the property operators care about, because it is the outage the tenant experiences. Three techniques shrink it, and each has a limit.

Dirty tracking on the device side reduces the guest pages that must be transferred in the final pass. If the device’s MMU can report which pages were written since a mark, the pre-copy loop can target those pages and converge. Without device-side dirty tracking, the device’s writes are invisible and every page the device touched must be assumed dirty, which for a tenant whose working set is its whole buffer means no convergence at all.

Incremental capture of device memory reduces the device-side transfer. A tenant’s device memory is typically allocated in large blocks whose contents change at a rate determined by the workload. If the driver can track which blocks changed, the capture can be incremental in the same way as guest memory pre-copy. This requires the driver to know the semantics of the allocations, which is why it is a driver feature rather than a framework feature.

Deferred restoration reduces the perceived stop. If the device can be brought up with its memory unmapped and faulted in on demand, the destination can start the guest before all of the device memory has arrived. This works when the tenant’s first actions touch a small part of its working set, which is true for a process that is starting up and false for one that resumes a long-running computation.

The limit that none of these remove is the fourth state domain. A kernel in flight cannot be captured without hardware support, so a design must either wait for it or preempt it. Waiting is bounded by the kernel’s duration, which is unbounded from the hypervisor’s point of view. Preemption bounds it at the granularity the hardware supports, and the cost of preemption is the state that must be saved, as described in the context switch analysis.

Destination constraints

A migrated guest must land on a device that can accept it. The constraints are stronger than for a CPU-only guest because device memory contents must be placed at addresses the guest believes are valid.

If the guest’s driver addresses device memory through guest-physical addresses that the host maps to device memory, then the placement is the host’s decision and the destination can choose different device addresses. This is the design that makes migration tractable: the guest never learns a device address, so the host can relocate. If the guest’s driver holds device addresses directly, the destination must provide the same addresses, which means the same device model, the same memory layout, and the same BAR placement.

The other constraint is the device’s capability set. The destination must support the features the guest’s driver has already programmed: the same queue count, the same memory size or more, the same partitioning shape. A migration that lands on a device with fewer queues fails at the point the driver reads the capability, which is after the guest has resumed and therefore after the outage has begun. Checking capability compatibility before the transfer starts is the only way to avoid that failure mode, and it requires the host to know the guest’s programmed configuration, which it does only for a mediated device.

The failure that matters most

The interesting failure is not a failed transfer but a transfer that succeeds and leaves the source’s device believing it still owns the tenant. If the source’s device was quiesced and its translations invalidated, its state is inert and it can be reset for reuse. If either step was skipped, the source device may still hold cached translations to frames that have been reallocated, and the next tenant on that device inherits a device capable of writing into an unrelated guest’s memory.

This is why teardown on the source is not an optimisation opportunity. It must complete before the destination is allowed to run the guest, and it must be verified, not merely issued. The verification is the same one that applies to ordinary device teardown, and the reason it is easy to get wrong is that it happens on the failure path, where the pressure is to release resources quickly.

The second-order failure is a partial restore at the destination: device memory installed, translations installed, and architectural state not yet restored. A guest resumed in that condition will program the device from a state that is half-old, and the resulting behaviour depends on which registers were restored. The fix is the ordering: architectural state is restored last, and the guest is not resumed until it is complete.

What migration buys and what it costs

Migration buys the ability to drain a host for maintenance, to rebalance tenants across a pool, and to move a tenant away from a failing device. The cost is that every tenant must be migratable for the pool to be uniform, and a tenant that holds a device with no migratable state cannot be part of the pool. The designs that support migration are therefore the ones that give up direct access, and the designs that give direct access are the ones that give up migration. Interference controls are the other half of the same operational story: a pool that cannot migrate must partition, and a pool that partitions cannot rebalance.