The Stop-and-Copy Boundary in Live Migration

A virtual machine can be moved between hosts while it runs because the hypervisor, not the guest, owns the mapping from guest physical addresses to machine memory. Nothing inside the guest refers to a host, a NUMA node, or a physical address that the guest can observe. That indirection is what makes relocation possible at all: the guest’s view of its own memory is a fiction the hypervisor can re-create anywhere.

What the indirection does not give is a moment when the guest’s state is small. A running guest holds a full memory image, the register file of every vCPU, the state of every emulated or paravirtualized device, and in-flight DMA. Moving all of it while the guest keeps executing means the destination must eventually receive a consistent set of those four things. The stop-and-copy boundary is the instant at which consistency is imposed: the vCPUs stop, the remaining divergence is copied, and execution resumes on the other side.

Downtime is not migration time

Two different intervals get conflated. Total migration time runs from the decision to migrate until the destination is serving and the source has released its resources; it can be seconds or minutes. Downtime is the sub-interval during which the guest cannot make forward progress, because its vCPUs are stopped and the guest-visible clock is held still.

The distinction matters because almost every technique in this cluster buys a shorter downtime by spending more total time. Pre-copying memory repeatedly while the guest runs, compressing the stream, or deferring page transfer to the destination all reduce the frozen window at the cost of bandwidth, CPU on both hosts, or a longer exposure to failure. The design question is never “how do we migrate faster” in the abstract; it is “how much of the state can be transferred before the freeze, and what is irreducible inside it”.

A useful decomposition is:

• Irreducible state at the boundary: the vCPU register files, the device state blobs, and the pages that were written since the last copy pass. All of this must cross the wire while the guest is stopped.
• Deferrable state: pages that are clean at the boundary. They can be sent before or after, or fetched on demand.

Only the first term sets the downtime floor. This is why a migration of a guest with a large but mostly idle memory image can be nearly transparent, while a guest with a small image that rewrites it constantly cannot.

Quiescing the vCPU set without losing device state

Stopping vCPUs is the easy part in isolation: the hypervisor issues an inter-processor interrupt or waits for each vCPU thread to reach a VM exit, then parks it. The hard part is everything that continues to mutate guest memory while the vCPUs are not running.

Devices are the usual offender. A virtio-net device with a live queue can still be reading descriptors the guest wrote before it stopped, and a passed-through device can DMA into guest memory with no hypervisor involvement at all. The freeze therefore has to proceed in an order that drains outstanding work before the final memory copy begins:

1. Stop the vCPUs and save their register state, including control registers, MSRs, and the architectural state of any nested paging structures.
2. Quiesce device emulation: stop the event loop that services virtio queues, disable queue notifications, and let any in-flight descriptor processing complete.
3. Drain DMA. For emulated devices this means waiting for outstanding requests the hypervisor itself issued; for assigned devices it means the device cannot be migrated at all and must be unplugged before the freeze.
4. Save device state: queue base addresses, indices, and the device-specific configuration space that the guest wrote.
5. Copy the residual dirty set and the device state to the destination.
6. Restore on the destination in reverse order, then start the vCPUs.

The ordering is not cosmetic. A device state blob captured while the guest could still program a queue descriptor would describe a queue whose contents changed afterwards; the destination would restore a device whose indices disagree with memory.

/* Shape of the freeze, not a real API: the invariant is the ordering. */
static int vm_freeze(struct vm *vm)
{
    if (pause_vcpus(vm))              /* each vCPU exits to the hypervisor */
        return -EAGAIN;
    save_cpu_state(vm);               /* registers, MSRs, FPU, nested state */
    device_quiesce_all(vm);           /* stop virtio event handling */
    if (device_drain_all(vm))         /* outstanding emulated I/O */
        return -EBUSY;
    save_device_state(vm);            /* queue indices, config space */
    return 0;                         /* memory still needs the residual copy */
}

Note what is missing: the memory copy. The freeze and the copy are separate steps because the copy may be large and its size determines the downtime. Everything above is bounded by the number of devices, not by guest RAM.

The residual copy and the commit point

At the moment the vCPUs stop, the destination holds some prefix of the memory image: whatever previous passes delivered. The source holds the authoritative image. The residual copy sends the difference — the pages marked dirty since the last pass — plus the device blobs.

Until that copy completes and the destination acknowledges it, the source is still the only correct copy of the guest. This is the commit point, and it is why a migration can be aborted safely before it. The source either keeps running (if the abort happens before the freeze) or thaws its vCPUs and resumes (if the abort happens during the freeze, provided no state has been discarded).

Once the destination has the residual set, two options exist, and they differ in what happens on failure:

• Source-retains-state: the source keeps its memory image intact until the destination reports that it is running. A failure on the destination can be rolled back by resuming the source, at the cost of holding double the memory footprint during the handover.
• Source-discards-state: the source frees pages as the destination acknowledges them. Failure after the commit point leaves no fallback. This is the mechanism behind post-copy migration, described in Post-Copy Migration and Demand Paging from the Source.

The residual copy size is what the guest experiences as downtime, and it is bounded below by the wire rate. A 4 GiB guest with a 1 Gbit/s link cannot move its full image inside a millisecond; that is roughly 34 seconds of raw transfer for the whole image, which is why the whole image is never sent inside the freeze. The freeze only ever carries the delta.

What the guest observes during the freeze

The guest does not see a scheduler gap. Its vCPUs simply do not run, and the clock sources it reads are virtualized, so the hypervisor controls what time appears to pass. Three clock behaviours matter:

• TSC: modern parts provide a stable, invariant TSC, but its value differs between hosts. The destination must offset the guest’s view so that the guest sees monotonic time across the move. A guest that reads the TSC in a tight loop around the freeze will observe a jump unless the offset is chosen to hide the gap.
• Emulated timers (HPET, PIT, local APIC timer): these are part of device state and are restored with the rest of it. A timer that was due to fire during the freeze fires immediately after resume, which is correct but can produce a burst of expiries.
• Paravirtual clock: the pvclock structure carries a per-vCPU offset and a version counter; the destination republishes it. Guests that cache the offset without re-reading the version can compute wrong time, which is why the sequence counter exists.

Network-visible effects are separate and are covered in Network Path Continuity Across a Migration. For the guest’s own view, the contract is narrow: time does not go backwards, timers do not fire early relative to guest-visible time, and no memory the guest wrote before the freeze is lost.

Why the freeze cannot be made arbitrarily short

The floor is the sum of the residual copy and the device state save, both serialized while the vCPUs are stopped. Shrinking it requires shrinking the residual set, which is the subject of dirty tracking and convergence, or overlapping the copy with execution, which post-copy does by letting the destination fault pages back from the source.

There is a harder limit that no amount of engineering removes: the guest must be stopped at some point for its state to be a snapshot. A memory image copied while the guest runs is a smear of different instants, and restoring it produces a guest whose data structures are internally inconsistent. Every technique in this cluster is an argument about how small the smear-free window can be made, not whether it can be eliminated. Dirty page tracking, described in Hardware Dirty-Bit Tracking for Guest Memory, exists precisely to make the smear as small as possible before the boundary is crossed.

A second limit is failure exposure. A longer total migration time is a longer interval during which a host failure loses the guest, and a longer interval during which the source must keep its memory image and its network identity. Operators routinely cap migration time rather than minimize downtime, because an unbounded migration that never converges is worse than a briefly frozen guest. That cap, and the convergence condition behind it, is the subject of Dirty Page Rate and the Convergence Condition.