Serializing Device State Across a Migration

Memory is the large part of a migration but not the delicate part. Guest RAM is a flat byte array; copy it, and it is correct. Device state is a collection of registers, queue positions, timers, and pending interrupt lines whose internal representation belongs to the hypervisor, not to any standard. Reconstructing it on the destination requires both sides to agree on a format, and the agreement has to survive hypervisor versions, configuration differences, and devices that are partially through an operation when the freeze arrives.

A device restored with the wrong queue index will either re-deliver descriptors the guest already consumed or skip descriptors it has not. A device restored with a pending interrupt that was already acknowledged will inject a spurious interrupt into a guest that has moved on. Neither failure is detected by anything; the guest simply misbehaves later.

What counts as device state

For a paravirtualized device such as virtio, the state that must survive a move is smaller than it first appears, because most of the device’s operation is defined by memory the guest owns:

• Feature negotiation result: the subset of feature bits the guest accepted at driver load. The destination must present the same set; if it presents more, a guest driver that re-reads the feature word mid-operation could behave differently, and if it presents fewer, a driver using a feature it negotiated would break.
• Device status field: the driver’s progress through the virtio initialisation sequence, including the DRIVER_OK bit.
• Queue state: for each virtqueue, the descriptor table base address, the available ring base, the used ring base, and — critically — the last-avail-idx the device has observed and the last-used-idx it has published. These are indices into guest memory, and they are the device’s cursor through the ring.
• Device-specific configuration: block device capacity and geometry, network device MAC address, and any per-device counters the guest can read.
• Pending interrupt state: whether an interrupt is asserted and not yet acknowledged.

For an emulated (non-paravirtual) device such as an IDE controller or a legacy VGA adapter, the state is the full register file, including internal state machines mid-transfer. An IDE controller in the middle of a multi-sector DMA has a sector counter, a buffer position, and an active command; restoring it means restoring all of that, and the destination must continue the transfer from the same byte.

For an assigned device, the state is inside the hardware and generally cannot be extracted. This is why device assignment and migration are mutually exclusive in most configurations: the device must be unplugged (hot-unplugged from the guest) before the memory migration begins, which changes the guest’s hardware and is visible to it.

The queue index is the whole protocol

The virtio split-ring design makes the queue state the most important part of the migration. The driver adds buffers by incrementing the available ring’s index; the device consumes them and increments the used ring’s index. Each side tracks its own view of both indices.

The device’s saved state is its last-avail-idx, which tells the destination how many buffers it has already consumed, and its last-used-idx, which tells it what it has already published. If the destination restores a last-avail-idx that is too low, it will re-consume buffers the source already processed, which for a block device means reading or writing the same sector twice, and for a network device means transmitting a frame twice. If it is too high, buffers are skipped and the guest’s data is silently dropped.

The subtlety is that these indices are read from guest memory, not stored in the device. The device reads the available ring’s index field from guest RAM on each notification. The saved value is a cache of what the device last saw, and it must be consistent with the memory image the destination receives. If the guest advanced the available index and the device had not yet observed it, the destination’s cached value is behind guest memory, which is correct: the destination will observe the advance on its next notification. If the device had observed the advance but not yet written the used ring entry, the destination’s cache is ahead of the used ring in memory, and it must write the used entry during restore, not wait for a notification that has already been consumed.

That last case is why queue state cannot be saved as a simple register dump. The save must capture the device’s progress through any in-flight descriptor, and the restore must complete that descriptor’s bookkeeping in the used ring before the guest can be resumed.

Interrupts are state, not events

An interrupt line in a virtual device is not an edge in the host; it is a level that the guest’s interrupt controller observes. Saving the line’s level is not enough, because the guest’s local APIC holds a pending-interrupt bit per vector, and the PIC or IOAPIC holds its own latched state.

A migration that saves the device’s “interrupt asserted” flag but not the interrupt controller’s pending vector will either lose the interrupt or deliver it twice, depending on whether the destination’s controller starts empty. Saving the controller state is part of the CPU or platform state blob, not the device blob, and the ordering between the two restores matters: the controller must be restored before devices are allowed to raise new lines, or a device that restores with its line asserted will inject into an empty controller and the vector will be lost.

There is a related issue with interrupt coalescing. Some devices defer interrupts to reduce exit rates, holding a pending flag until a timer fires or a queue drains. That deferral state is part of the device, and a migration that drops it can either deliver a burst of coalesced interrupts on the destination or lose the pending ones. Neither corrupts memory, but both disturb the guest’s timing.

The format problem

Device state is hypervisor-internal. There is no standard wire format for it, which means migration compatibility is a property of the hypervisor pair, not of the guest. Two mechanisms make this workable:

Versioned state sections. The stream is organised as named sections, each with an identifier and a length. A device writes its state into its section; the destination looks up the section by name and hands the bytes to the matching device. Unknown sections are skipped (their length tells the reader how far to jump), which is what allows a newer source to migrate to an older destination: the older destination ignores the sections it does not understand. The reverse direction fails unless the newer destination can synthesise the missing sections, which is why compatibility is typically one-directional across versions.

Device model matching. The destination must instantiate the same device model with the same configuration. A virtio-net device with two queues on the source cannot be restored onto a destination configured with one queue, because the queue state section has two entries and the device has one. Implementations therefore compare a machine type and device configuration before starting, and refuse the migration rather than attempting a partial restore.

The refusal is the right behaviour and is often the only safe one. A device model that silently accepted a mismatched configuration would produce a guest that boots but whose driver indexes a ring that the device does not implement.

State that resists serialization

Some device state has no meaningful representation outside its host. A passthrough network adapter’s internal filter tables, a GPU’s command processor state, an SR-IOV virtual function’s hardware queues: these live in silicon and the hypervisor can only read or write them through device-specific interfaces, if at all. The general answer is that such devices are not migrated; the guest is told they were removed.

For emulated devices the analogous problem is timer state that depends on host time. A device timer armed for 50 ms from now, saved as an absolute deadline in host nanoseconds, restores on a destination whose monotonic clock has a different origin. The saved value must be relative to a common reference, or the destination must convert it. Getting this wrong produces a timer that fires immediately (deadline already past) or never (deadline far in the future), and the guest’s driver may interpret either as a device hang.

/* Timer state must be saved as a remaining interval, not an absolute tick. */
struct timer_state {
    uint64_t remaining_ns;   /* relative to the freeze instant */
    uint8_t  active;
};

static void save_timer(struct vtimer *t, struct timer_state *out)
{
    out->active = t->armed;
    out->remaining_ns = t->armed ? deadline_sub(t->deadline, now_ns()) : 0;
}

static void restore_timer(struct vtimer *t, const struct timer_state *in)
{
    t->armed = in->active;
    if (t->armed)
        t->deadline = now_ns() + in->remaining_ns;  /* rebase to this host */
}

The same rebasing applies to the guest’s own clock sources, discussed in The Stop-and-Copy Boundary in Live Migration, and to the network device’s link state, which the destination’s virtual switch must adopt before the guest transmits, as described in Network Path Continuity Across a Migration.

Where device state dominates the downtime

For a guest with a large memory image, device state is a rounding error: a few kilobytes per device against gigabytes of RAM. For a small guest, it can dominate. A guest with 256 MiB of RAM and a dozen emulated devices may spend more of its freeze serializing devices than copying pages, because device serialization is inherently serial and cannot be parallelized across the wire the way page transfer can.

That asymmetry explains why device count, not guest size, is the practical limit on how short a freeze can be made. Reducing the number of emulated devices, preferring paravirtualized ones with small state, and avoiding device models with deep internal pipelines are the levers. None of them are visible from inside the guest, which continues to see the same hardware either way.