Abort Paths and the Blackout Window

Every migration has a point after which the source can no longer take the guest back. Before that point, a failure on either side is recoverable: the source still has a complete, running guest, and the operation can be abandoned with the guest continuing where it was. After that point, the guest’s only complete state is on the destination, and a failure there is a failure of the guest.

The boundary between those two regimes is the commit point, and its placement is a design decision with consequences that are easy to state and hard to get right. Placing it late keeps rollback available for longer but leaves the source holding two copies of the memory image for the whole migration. Placing it early shortens that exposure but makes the guest’s survival depend on the destination from an earlier moment.

Four failure points, three outcomes

A migration can fail at distinct stages, and the recovery differs at each:

Before the freeze. Memory is being copied while the guest runs. If the destination becomes unreachable or the link drops, the source simply stops sending. The guest never noticed. The source frees whatever it staged on the destination and continues. This is the cheapest failure to handle because nothing guest-visible has changed.

During the freeze. The vCPUs are stopped and the residual is in flight. A failure here leaves the guest frozen on the source with a partial copy at the destination. The source must thaw its vCPUs and resume. The correctness requirement is that the source’s memory image has not been modified since the freeze — which is true, because the vCPUs are stopped — and that no device state has been discarded. If the source has already sent the only copy of some device state, resuming is unsafe.

After the destination has the state but before it reports success. The destination has a complete image but has not yet confirmed that it is running. The source cannot discard its image, because the destination might fail during its own resume sequence. The source waits for an acknowledgement; if none arrives within a timeout, it must decide whether to resume the guest itself. Resuming while the destination may also be running produces two copies of the guest, which is worse than either failure alone.

After the acknowledgement. The source’s state is redundant. If the destination fails now, the guest fails. The source’s only remaining job is to release resources.

The third case is the one that needs a tie-break. A timeout cannot distinguish “the destination is dead” from “the destination is running the guest and the acknowledgement was lost”. Both look identical to the source. The mechanisms that resolve it are the same ones used for storage fencing: a shared lock or lease that only one side can hold, or a generation counter that the destination must present to prove it is the current owner. Without such a mechanism, a network partition during the handover is a split-brain by construction.

The freeze as a blackout budget

Downtime is the interval during which the guest makes no progress. It is not exactly the freeze, because the guest’s perception of time depends on the clock sources it reads and the timers it has armed. Three intervals compose:

• vCPU stop to vCPU resume: the interval in which no guest instruction executes. This is what a guest measures with a tight loop reading the TSC.
• Guest-visible time gap: the interval the guest believes passed, which can be shorter than the real interval if the destination rebases the clock to hide part of it, or longer if the destination’s clock is behind.
• Timer expiry burst: timers that came due during the freeze fire immediately after resume. The guest sees no elapsed time but sees a burst of expiries, which for a guest with many armed timers is a burst of interrupt handling that delays forward progress.

The blackout budget for a migration is usually stated as the first interval, because it is the one the hypervisor controls and measures. But a guest’s failure detectors use the second and third: a cluster heartbeat that measures guest-visible time sees the rebased value, while a guest that measures wall time from an external source sees the real gap. This is why hiding the gap in the clock does not hide the migration from the guest’s peers.

Source teardown and the resources it holds

After the destination acknowledges, the source releases: guest memory, device backends, network attachments, and any storage reservations. Each release has a failure mode.

Memory release is the easy one. Once the destination has confirmed that it has the complete image, the source can free it. The confirmation must be positive, not inferred from the absence of errors; a source that frees memory on a timeout risks the destination needing a page that no longer exists.

Network release is the dangerous one. The source’s tap device or bridge port still exists and may still be configured with the guest’s MAC. If the source’s host stack answers an ARP request for the guest’s IP, peers learn a stale mapping; if it receives a frame for the guest’s MAC, it may drop it, which is harmless, or it may respond with an ICMP unreachable or a TCP reset, which is not. The teardown must remove the attachment before the guest’s peers can be confused by it, and the sequencing is discussed in Network Path Continuity Across a Migration.

Storage release is the one with the longest tail. A shared LUN’s reservation must be released so the destination can take it, and the source’s block device must be closed so its page cache does not flush stale data. If the source is unresponsive, the release does not happen, and the destination’s attempt to take the reservation fails. The operational answer is preemption: the destination forcibly breaks the source’s reservation after a timeout. Preemption is safe only if the source is genuinely unable to write, which is what the fencing mechanism asserts.

Rollback is not free

A migration that aborts during the freeze requires the source to thaw and continue. That is only possible if the source’s state is intact, and several things can make it not intact:

• The source may have discarded pages it already sent, in a scheme where the destination acknowledges page receipts and the source frees them. Then the source’s image has holes, and thawing produces a guest with missing memory.
• The source may have reset its devices to capture their state. A device reset discards in-flight operations, which is fine if the guest is going to resume on the destination, and wrong if it resumes on the source.
• The source may have torn down its network attachment in anticipation of the handover. Thawing then produces a guest with no connectivity.

The general rule is that a source can abort cheaply up to the point where it starts destroying its own state, and after that point the only safe completion is forward. Implementations therefore order the destructive steps as late as possible: device reset and network teardown happen after the destination has acknowledged, not before.

There is an exception for the freeze itself. During the freeze, the source’s vCPUs are stopped and its devices are quiesced, but nothing has been destroyed. An abort during this window is safe and cheap: thaw and resume. This is why the freeze is a good place to put a final validation — the destination can confirm it has all the sections it needs before the source commits, and the source can still back out.

Validating before committing

A migration that fails validation after the guest has moved is unrecoverable, so the validation belongs before the commit point. The checks that matter:

• Machine type and device configuration: the destination must be able to instantiate the same devices with the same parameters. A mismatch here cannot be fixed after the fact.
• Section completeness: every section the source wrote must either be consumed by the destination or explicitly ignorable. A required section that the destination does not recognise is a hard failure.
• Memory availability: the destination must have the guest’s full memory allocated before the guest resumes. Post-copy is stricter about this than pre-copy, as described in Post-Copy Migration and Demand Paging from the Source.
• Storage claim: the destination must hold the reservation or the disk attachment before the guest can write.

A destination that fails any of these can still refuse, and the source resumes with the guest unaware. The cost is the wasted transfer, which is the price of validating late rather than early. Validating early is cheaper but incomplete, because some properties — like whether the destination can actually map the guest’s memory — are only knowable once the attempt is made.

The asymmetry between pre-copy and post-copy

Pre-copy abort is symmetric: either side can fail before the commit and the guest survives on the source. Post-copy abort is not. Once the destination has started the guest, the source cannot take it back, because the destination’s guest has already modified its memory; the two images have diverged, and there is no way to reconstruct a consistent state from them.

The only recovery from a post-copy failure is to keep the source’s memory available long enough for the destination to fetch what it needs, which is exactly the dependency that post-copy creates. If the source fails, the guest cannot be recovered at all. That is the price of the bounded freeze, and it is why post-copy is used for workloads where a long freeze is unacceptable and a short window of non-recoverability is tolerable — not as a general replacement for pre-copy.