Post-Copy Migration and Demand Paging from the Source

Pre-copy migration is an argument that the guest’s dirty rate will fall below the link rate and the residual will shrink to something that fits in a freeze. When that argument fails — a guest that rewrites its whole working set continuously, a link that is slower than the guest’s write bandwidth — the migration either never converges or is abandoned after a timeout.

Post-copy inverts the order. The destination receives the vCPU state and a small amount of memory, starts the guest, and then faults in the remaining pages from the source as the guest touches them. There is no convergence condition to satisfy because there is no requirement that memory arrive before the guest runs. The freeze is bounded by the size of the register and device state, which is kilobytes, not gigabytes.

The cost moves elsewhere: every page the guest touches that has not arrived becomes a page fault whose handling requires a network round trip, and the source must stay alive, reachable, and holding the memory image until the last page has been fetched.

The fault path

A guest page that has not been transferred is not present in the destination’s nested page tables. When the guest accesses it, the CPU takes a nested page fault, the hypervisor’s fault handler runs, and it must obtain the page. In post-copy mode, obtaining it means asking the source.

The request travels over the same migration channel, in the reverse direction. The source looks up the guest physical address in its own memory, sends the page, and the destination maps it into the guest’s nested page tables and resumes the vCPU. The guest’s access then completes.

The latency of that path is the sum of the fault exit, the request transmission, the source’s handling, the page transmission, and the destination’s mapping. On a local network with a page-sized transfer, that is dominated by the round trip and the fixed costs of the fault and the mapping — hundreds of microseconds to a few milliseconds, against a normal memory access of tens to hundreds of nanoseconds. Every faulted page is therefore thousands of times more expensive than a resident one.

The saving grace is that faults are rare in steady state if the pages are fetched in a sensible order. A guest that touches a page once and then works within it pays one fault for a long stretch of execution. A guest that walks a large data structure pays one fault per page of the structure, which can be thousands of faults in a tight loop.

Pre-paging, and why the order matters

The destination cannot know which pages the guest will touch next, but it can guess. Three sources of information are available:

• The source’s own recent activity: pages the source’s guest was actively using before the freeze are likely to be used again soon. If the source can report access bits from its nested page tables, the destination can prefetch that set.
• The transfer order of a pre-copy phase: a migration that ran some pre-copy passes before switching to post-copy has already identified the pages that were not dirty, which are the ones unlikely to be hot. Fetching the dirty ones first is the wrong order; fetching the clean ones in bulk while demand-faulting the hot ones is right.
• Readahead on fault: when the guest faults a page in a contiguous region, fetching its neighbours speculatively is cheap because the requests can be batched and the source can serve a run of pages in one message. The risk is fetching pages that are never used, which wastes bandwidth but does not hurt correctness.

The literature on this describes pre-paging as adaptive: the destination tracks fault addresses and, when it sees a pattern, prefetches ahead of the pattern. A guest faulting sequential addresses gets large prefetch windows; a guest faulting scattered addresses gets none. The mechanism is a heuristic and it can be wrong in both directions, but the wrong direction that hurts is over-prefetching on a congested link, because the prefetch competes with demand faults for the same bandwidth and can delay the faults that actually block the guest.

The source becomes a server with a lifetime

In pre-copy, the source’s memory image can be discarded as soon as the destination acknowledges the final residual. In post-copy, the source’s image is the backing store for every page not yet fetched, and it must remain intact and correct until the destination has them all.

That changes what the source is allowed to do:

• It cannot start another guest in the same memory.
• It cannot free the pages to a host-level allocator.
• It cannot be shut down, rebooted, or migrated itself.
• It must keep a network path to the destination, with enough bandwidth to serve faults.

The lifetime is bounded by the fault rate, not by a timer. A guest that never touches some of its pages — a large heap that was allocated and never used, a set of zero pages that the destination already knows about — leaves those pages on the source forever unless something actively sweeps them. Implementations therefore run a background pass that copies the remaining pages in bulk after the guest is running, competing with demand faults for bandwidth but with lower priority. The migration is complete when that pass finishes, and only then can the source release the image.

The failure mode is severe: if the source dies before the sweep completes, the pages it still holds are lost, and the guest on the destination faults on memory that no longer exists. There is no recovery from that inside the guest; the destination can only terminate it. This is the fundamental trade of post-copy — it converts a convergence problem into an availability dependency, and the dependency lasts longer than the migration’s visible duration.

When post-copy wins and when it loses

The case where post-copy is clearly better is a guest whose dirty rate exceeds the link rate. Pre-copy cannot finish such a migration at all; post-copy finishes it with a bounded freeze and pays fault latency. A guest running a memory-resident database with a large hot set, or a guest whose working set is being actively rewritten, is the target workload.

The case where it loses is a guest whose memory is touched broadly and randomly after resume. Every fault is a round trip, and if the guest’s execution is dominated by memory accesses that miss, the slowdown compounds: the guest runs slower, which means the migration takes longer, which means more pages are faulted. The pathological version is a guest whose entire working set is larger than what can be held locally and is accessed uniformly, where post-copy produces a fault storm that never abates.

There is a middle case that is common: a guest with a large but mostly idle memory image. Post-copy freezes almost instantly, the guest resumes, faults a small hot set, and the background sweep copies the rest while the guest runs. The fault rate is low, the freeze is short, and the total time is dominated by the sweep at whatever bandwidth is left over. This is the case post-copy was designed for, and it is why the technique is attractive for guests that are large but not uniformly busy.

A third consideration is the destination’s memory pressure. Post-copy requires the destination to have room for the pages as they arrive, plus the pages it maps for the guest’s use. If the destination is overcommitted, it may need to evict pages it has already fetched, which is impossible in post-copy mode because those pages are the guest’s only copy. The destination must therefore reserve the full guest memory before starting, which removes the overcommit flexibility that pre-copy allows.

Interaction with the guest’s own paging

A guest that pages its own memory — swapping to its virtual disk — behaves differently under post-copy than under pre-copy. Under pre-copy, the guest’s swap activity dirties memory, which inflates the dirty rate and works against convergence. Under post-copy, a guest that faults on its own swap file generates two faults: one for the page table entry, one for the swapped page, and the second is a disk I/O on the destination.

The interaction is worse than additive when the guest’s swap device is also being migrated. A page fault that requires the destination to fetch a page from the source, which itself requires the source to read from a disk that is being copied, produces a chain of dependencies whose latency is the sum of all of them. Implementations avoid this by not combining post-copy memory migration with block migration of the same guest, or by ensuring the swap device is on shared storage.

The freeze it actually achieves

The stop-and-copy window in post-copy contains the vCPU state, the device state, and a minimal bootstrap set of pages that the guest needs to execute its first instructions after resume — typically the pages containing the guest’s resume vector, its page tables, and whatever the guest touches in its first few microseconds.

That set is small and bounded, but it is not zero, and getting it wrong produces a guest that faults immediately on resume, which is correct but slow. The alternative to guessing is to let the guest fault, which means the first instructions after resume are the most expensive ones. Implementations generally accept a few dozen faults in the first milliseconds rather than trying to predict the bootstrap set.

The mechanism that makes the freeze short is the same one that makes the source a dependency: the destination is allowed to start the guest with an incomplete memory image. Everything in The Stop-and-Copy Boundary in Live Migration about the ordering of the freeze still applies — devices must be quiesced, state captured, and the commit point respected — but the residual copy inside the freeze is replaced by a promise to fetch later, and the promise is what Abort Paths and the Blackout Window rules out rolling back.