Dirty Page Rate and the Convergence Condition

Pre-copy migration sends the guest’s memory while the guest runs, then repeats, sending only what changed since the previous pass. The scheme is attractive because it needs no change to the guest and keeps the freeze short. It also has a termination condition that is easy to state and easy to violate: the guest must dirty memory more slowly than the migration stream can carry it away.

When that condition holds, each pass is smaller than the last and the sequence converges geometrically. When it fails, the passes stay the same size forever and the hypervisor must eventually give up and freeze the guest anyway, having moved the entire image several times over.

The recurrence behind the pass sizes

Let $M$ be the guest’s resident memory, $R$ the effective transfer rate of the migration stream in bytes per second, and $D$ the rate at which the guest dirties memory, also in bytes per second. Pass zero sends all of $M$ and takes $M/R$ seconds. During that interval the guest dirties approximately $D \cdot M/R$ bytes.

If $W_n$ is the volume sent in pass $n$, the next pass carries roughly the dirtying that happened during pass $n$:

$$W_{n+1} \approx W_n \cdot \frac{D}{R}$$

Define the ratio $r = D/R$. The sequence is geometric with ratio $r$, so the total volume is

$$W_{\text{total}} = M \cdot \frac{1}{1-r}$$

and the number of passes needed to bring the residual below some threshold $W_{\min}$ is about $\log(W_{\min}/M)/\log r$. Both expressions fail at $r \ge 1$: the total diverges and no pass count suffices.

Two consequences follow directly. First, migration cost is not proportional to memory size alone; a guest with $r = 0.9$ moves ten times its image, while the same guest at $r = 0.5$ moves twice its image. Second, the pass count grows only logarithmically in the ratio, so it is the approach to $r = 1$ that hurts, not the memory size: at $r = 0.5$ a residual of one percent of the image is reached in about seven passes, while at $r = 0.9$ the same threshold takes roughly forty. Halving $r$ from 0.9 to 0.45 cuts the total volume by a factor of about five and brings the pass count back under ten.

Why the measured dirty rate exceeds the guest’s write rate

The $D$ in the recurrence is not the guest’s application write throughput. It is the rate at which pages are marked dirty, multiplied by the page size, and the difference between those quantities is often large.

A dirty bitmap has one bit per page, typically 4 KiB. Any write to any byte of a page marks the whole page, so a guest that updates one 8-byte counter per page in a large array dirties 4 KiB for every 8 bytes written: an amplification factor of 512. Database workloads that touch index pages randomly, and guests running many small allocations across a heap, produce the same effect. The migration stream must carry all 4 KiB.

The hypervisor has no way to distinguish the written bytes from the rest of the page without leaving the bitmap regime for something finer, and finer tracking costs either hardware support or a trap on every write. Both are more expensive than the amplification they would avoid. This is the reason huge pages make migration harder: a 2 MiB page turns a single byte write into 2 MiB of transfer, an amplification factor of 262,144 relative to that write.

There is a second inflation: the bitmap itself must be reset between passes. Clearing the bitmap for a 4 GiB guest at 4 KiB granularity touches 128 KiB of tracking state, which is small, but the clear must be ordered against concurrent guest writes so that no dirty bit is lost. The usual implementation clears the bitmap in chunks and re-reads it immediately before sending, which means a page written during the clear is either reported in this pass or the next, never dropped.

Choosing the cutoff

A pre-copy loop needs a stopping rule, because even a converging migration can be slow if $r$ is close to one. Implementations combine several conditions, and the first one to fire ends the pre-copy phase:

• Residual threshold: stop when the dirty set falls below a configured volume. The remaining pages plus device state then cross the wire during the freeze, so the threshold is effectively a downtime budget expressed in bytes.
• Pass limit: stop after a fixed number of iterations regardless of progress, converting an unconverging migration into a stop-and-copy one.
• Time limit: stop when total migration time exceeds a deadline, which protects the source from holding two copies of the memory image indefinitely.
• Bandwidth floor: stop when the achievable rate collapses, typically because the guest’s own network traffic or a competing migration is consuming the link.

The residual threshold is the one that maps directly onto guest-visible behaviour. If the destination can absorb the residual at the observed rate, the freeze lasts residual/$R$ plus the device state save. A threshold chosen without reference to the link rate produces a downtime that varies with link congestion, which is why the parameter is better expressed as a time budget than as a page count.

Measuring the rate honestly

Estimating $D$ from a single pass is unreliable because guest write behaviour is bursty. A pass that happens to overlap a checkpoint flush sees a dirty rate several times the steady-state value; a pass that overlaps an idle period sees almost nothing. Using the last pass alone as the predictor makes the stopping decision oscillate.

Practical estimators smooth over several passes, and some implementations track the trend: if pass sizes are decreasing by a healthy factor, continue; if they have flattened, the sequence is not converging and the cutoff should fire. This is a heuristic, and it fails in a specific way: a guest whose write rate is genuinely falling, but slowly, looks flat over three passes and gets frozen early, paying a longer downtime than necessary. The inverse error is worse, because continuing a non-converging migration consumes bandwidth without bound.

There is also an interaction with the transfer mechanism itself. If the migration stream shares a congestion-controlled path with guest traffic, sending faster increases loss and reduces the achieved rate, so $R$ is not a constant the sender controls. RDMA or a dedicated link makes $R$ predictable; a shared TCP path does not, and the estimate of $r$ then fluctuates for reasons unrelated to the guest.

When convergence is the wrong goal

The recurrence assumes the only way to finish is to shrink the dirty set below a threshold. Post-copy migration abandons that assumption: it freezes the guest after a single pass, starts it on the destination, and fetches the pages it did not send on demand from the source. The dirty set is then irrelevant, because pages that are still on the source are faulted in when touched. That inverts the failure mode: instead of an unbounded number of passes, the cost appears as page-fault latency inside the guest, and the source must remain alive and reachable until every page has been fetched. The mechanics are in Post-Copy Migration and Demand Paging from the Source.

A hybrid exists for guests that will not converge: run a bounded number of pre-copy passes to move the pages that are unlikely to be rewritten, then switch to demand paging for the rest. The benefit is that the pages most likely to be touched again are already local, which reduces fault traffic. The cost is that the switch decision has to be made from the same noisy rate estimate, and a bad decision leaves the guest faulting heavily against a remote source.

Both approaches share a constraint that pre-copy alone hides: the source’s memory must remain intact and addressable after the guest has moved. That constraint is why a pre-copy migration that converges is the cheapest design and why every alternative exists to handle the case where it does not.

The amplification is the real lever

Because $D$ is measured in dirtied pages, the most effective way to make a migration converge is to reduce the number of distinct pages the guest touches per unit time, not the number of bytes it writes. Ballooning the guest to shrink its working set, encouraging the guest to coalesce its writes, or pinning a hot region of memory on the destination all change the page-touch pattern rather than the write volume.

The same observation explains why the size of the guest matters less than its behaviour. A 64 GiB guest running an idle service has a dirty rate near zero and converges in one pass; a 2 GiB guest running a memory-mapped write-heavy loop never converges at any link rate. The dirty bitmap, and the granularity at which it records writes, is what turns the second case into an unbounded migration. Hardware Dirty-Bit Tracking for Guest Memory covers how that bitmap is maintained without trapping every store.