Storage Handoff: Moving a Guest's Disk With It

Memory migration assumes the destination can reconstruct the guest’s RAM from a stream. Disks break that assumption in two ways: they are much larger than RAM, and they are the durable record that the guest’s correctness depends on. A guest resumed with a memory image that is a few milliseconds stale recovers; a guest resumed with a filesystem whose journal refers to blocks the destination does not have does not.

The design space has three points. Shared storage leaves the disk where it is and re-points the attachment, which is fast but requires the storage to be reachable from both hosts. Block migration copies the disk contents as part of the migration, which works without shared storage but is slow and needs a consistency story. A hybrid copies the disk while the guest runs, then converges on a delta, using the same structure as memory pre-copy but with much worse amplification.

Shared storage and what “shared” has to mean

The simplest handoff is no handoff. If the guest’s disk is a LUN on a SAN or a file on an NFS export reachable from both hosts, the destination opens the same backing store and continues. Nothing about the disk contents moves.

What must move is the claim on the disk. Two hosts with the same LUN open read-write will corrupt it, because neither knows about the other’s writes. The mechanisms that prevent this are reservations and leases:

• SCSI persistent reservations let a host register itself and take an exclusive reservation on a LUN; a second host’s I/O is rejected until the reservation is released or preempted. The migration sequence is: destination registers, source releases, destination takes the reservation. The window between release and take is a window in which neither host holds it, which is safe only because the guest is frozen during it.
• NFS leases and locks serve the same function at the file level. An NFS client holds a lease on a file it has open; the server can revoke it, and a second client cannot obtain conflicting state until the first releases or its lease expires.
• Fencing is the operational backstop: a mechanism outside the storage path that forcibly disables the source’s access, used when the source is unresponsive and cannot release cleanly.

The failure mode to design against is the source that believes it still owns the disk. If the destination takes the reservation while the source’s kernel still has the block device open and dirty page cache, the source will eventually flush those pages to a disk it no longer owns. Persistent reservations do not stop a host that is already past the SCSI layer; the write reaches the LUN and the destination’s filesystem sees blocks change underneath it. This is why the source must close or invalidate the device, not merely stop the guest.

Shared storage also constrains where the guest can move. The destination must have a path to the same LUN, which means the same fabric, the same zoning, and the same multipath configuration. Migrating across data centres with independent storage is impossible without copying, no matter how good the network is.

Copying the disk while the guest runs

When there is no shared storage, the disk has to move. The naive approach — stop the guest, copy the image, start the destination — has downtime proportional to the image size, which for a terabyte-scale disk is hours.

The pre-copy structure from memory migration applies, with a critical difference in amplification. A block device’s dirty tracking granularity is the cluster size the hypervisor uses for the copy, typically 1 MiB or larger for a raw image, and a guest filesystem writing one 4 KiB block marks the whole cluster dirty. The ratio of written bytes to transferred bytes can be a hundred to one, which is far worse than the memory case and makes convergence much harder to achieve.

Two properties make block migration tractable anyway:

Write ordering within a cluster is not needed, but cluster consistency is. The destination receives clusters in some order, and a cluster that was dirty in pass $n$ may be written again in pass $n+1$. As long as each cluster’s final state on the destination matches its final state on the source, the image is consistent. There is no requirement that intermediate states be consistent, because the guest is not reading the destination’s image during the copy.

The final pass must be atomic with respect to the freeze. The last dirty clusters plus the guest’s memory state must land before the guest resumes, or the guest will read a cluster that has not been updated. This is the same commit point as in memory migration, and it is the point at which the disk copy and the memory copy have to be coordinated: the destination cannot resume the guest until both its memory and its disk are consistent with the source at the same instant.

That coordination is the hard part. Memory and disk are copied by separate loops at different rates, so they reach their residuals at different times. The migration driver has to hold the disk loop at its final delta while the memory loop finishes, or freeze the guest once and copy both residuals inside the freeze. The latter is simpler and produces a downtime that includes the disk residual, which is why block migration with a large residual is visibly slower to complete than memory-only migration.

Dirty tracking for block devices

The hypervisor needs to know which clusters changed. Three mechanisms exist:

• Bitmap in the hypervisor: the block layer records writes to the image and sets bits in a bitmap at cluster granularity. This works for any image format the hypervisor understands and costs a small amount of memory plus a check on the write path.
• Changed Block Tracking in the image format: a copy-on-write image format can record which blocks have changed since a snapshot, and a migration can use that record instead of maintaining its own. The advantage is that the tracking persists across host failures; the disadvantage is that it is tied to the format.
• Array-side tracking: a storage array can report changed regions, which is the only option for a LUN the hypervisor cannot intercept writes on.

The choice affects what happens after a failed migration. If the tracking is in the hypervisor’s memory, a source crash loses it, and a resumed migration must recopy the whole image. If the tracking is persistent, the migration can resume where it stopped.

Consistency of the guest’s view

A guest that is frozen during the final copy sees a disk whose contents are whatever the source had at the freeze. Its filesystem’s journal, if it has one, describes writes that were in flight; on resume, the guest’s own recovery code replays or discards them. This is exactly the state a guest would be in after a power loss, and filesystems are designed to survive it.

The trouble is when the hypervisor’s write cache is involved. A guest that issued a flush and received completion believes the data is durable. If the hypervisor acknowledged the flush by writing to a host page cache that has not reached the destination, the guest’s belief is false after the migration. The fix is to treat a flush as a barrier that must be satisfied before the migration completes: either the data is on the destination, or the flush completion was never delivered to the guest. The second option is not available after the fact, so the first is the only correct one.

The same reasoning applies to the ordering guarantees of the device. A virtio-blk device that advertises a write cache must honour flush requests; if the destination cannot guarantee the same durability as the source, the device’s advertised feature set should not include the cache, and the guest’s driver will issue flushes more often. This is a case where the honest answer is to reduce the device’s capabilities rather than to promise durability the migration cannot deliver.

The copy is not the migration

A disk copy that completes is not a migration that has succeeded. The guest’s memory may reference blocks that the disk copy has already moved past, and the destination’s filesystem cache may hold pages that were valid against the source’s disk but are stale against the destination’s. The resume sequence must invalidate the guest’s caches or guarantee that the disk image is byte-identical at the instant of resume.

Byte-identical is achievable only if the final delta is applied inside the freeze. If the disk loop is allowed to continue after the memory loop finishes, the two are no longer describing the same instant, and the guest resumes with a memory image from time $t_1$ and a disk from time $t_2 > t_1$. Filesystems with journals survive this because the journal is on the disk and describes the gap; filesystems without journals, or applications that write to raw block devices, may not.

The practical ordering is therefore: converge both loops, freeze, apply both residuals, verify, resume. The verification step is the one that is usually omitted, and it is where the interesting failures hide. A cluster that the disk loop believed was clean but that the guest wrote after the belief was formed is exactly the case the dirty bitmap must catch, and it is the case a buggy bitmap silently loses.

For guests whose disks are on shared storage, none of this applies, which is why shared storage remains the default for anything that must migrate frequently. The cost is the fabric, and the benefit is that storage handoff becomes a lease transfer rather than a data copy. The memory side of the handoff, which has its own residual and commit semantics, is described in The Stop-and-Copy Boundary in Live Migration, and the failure ordering that decides whether a half-moved guest can be rolled back is in Abort Paths and the Blackout Window.