Network Path Continuity Across a Migration

A guest’s network identity is not its location. Its IP address and MAC address are guest state and travel with it; the switch port those frames leave from is host state and does not. The gap between the two is what breaks naive migrations: the destination starts sending frames with the guest’s MAC from a different port, and every forwarding table between the guest and its peers still believes that MAC lives somewhere else.

The failures come in three layers, and they need three different remedies. The switch’s forwarding database must learn the new port. The peers’ ARP caches must accept the new location without treating it as an address conflict. And the guest’s own TCP connections must survive a window in which its frames are silently dropped.

The forwarding database does not know the guest moved

An Ethernet switch learns where a MAC address lives by observing the source address of frames arriving on a port. The entry ages out on a timer whose duration is vendor-dependent and typically measured in minutes. Until the entry is replaced or expires, frames addressed to the guest are sent to the old port.

If the source host’s port is still active — it usually is, because the source host is still running and may host other guests — the switch keeps forwarding to it. The frames arrive at the old host, which has no matching guest, and are dropped. Traffic to the guest disappears until something forces the switch to re-learn.

Two mechanisms fix this, and they operate at different timescales:

• Gratuitous transmission: immediately after the guest resumes on the destination, the destination sends frames sourced from the guest’s MAC. A broadcast or a frame addressed to the switch itself is enough to make the switch re-learn. The classic form is a gratuitous ARP, an ARP request or reply that announces the sender’s own address mapping without anyone asking for it. RFC 5227 defines the probe and announcement procedure for IPv4 address claims; a migration announcement is the same packet shape used for a different purpose.
• Reverse learning on receipt: a switch that receives a frame from a MAC on a new port updates its entry immediately, regardless of the age of the old entry. This means one frame from the destination is sufficient, provided it reaches the switch. If the destination’s port is a trunk carrying many VLANs, the announcement must be sent on each VLAN the guest belongs to, because forwarding tables are per-VLAN.

The announcement has to happen after the guest’s device is live on the destination but before the guest is likely to send its own traffic, otherwise the first frames the guest sends are the ones that get lost. Some implementations send the announcement from the hypervisor’s own network stack on the guest’s behalf, which requires the hypervisor to be able to source frames with an arbitrary MAC — possible on a bridge or a tap device in promiscuous mode, not possible on a switch port with MAC filtering.

MAC filtering is the harder variant. A data-centre access port often permits only a small number of learned source MACs, and a migration moves the guest’s MAC from one port’s permitted set to another’s. If the destination port has not been configured to accept the guest’s MAC, the announcement frames are dropped as spoofed, and the guest loses connectivity permanently rather than temporarily. This is a provisioning problem, not a protocol one, and it is why fabric configuration for migration is usually a separate step from the migration itself.

ARP caches, conflict detection, and the announcement’s side effects

Peers hold ARP entries mapping the guest’s IP to its MAC, and those entries are cached for minutes. The MAC does not change during a migration, so peers’ entries remain correct in content; what changes is which switch port the MAC is reachable through. Peers therefore need nothing, provided the forwarding database is corrected.

That changes if the destination or the hypervisor sends an announcement that looks like an address conflict. RFC 5227 specifies that a host defending its address sends an announcement, and that hosts receiving an ARP probe for an address they already use may respond defensively. A migration announcement is unsolicited and should not be a probe; if it is sent as a probe (sender IP field zeroed), other hosts running conflict detection may treat the address as newly claimed and, in the worst case, log a conflict or defend it.

The correct shape is an unsolicited ARP reply with the sender’s IP and MAC filled in, broadcast to the local segment. Linux and other stacks also implement this as a “arping” style announcement from userspace; a hypervisor can do it internally without the guest’s participation, which is preferable because the guest may be in the middle of its resume sequence and not yet able to send.

There is a second-order problem with routers. A router on the segment caches the IP-to-MAC mapping for the guest and forwards frames to that MAC; the switch re-learns, so the router’s frames reach the guest. But the router’s own ARP entry has an age, and if the guest’s traffic stops long enough for it to expire, the router re-ARPs and the request is broadcast — which reaches the guest at its new port with no problem. The failure is only in the window before the switch re-learns, and the announcement closes that window.

The TCP connections that were mid-flight

A guest with established TCP connections is the case where a short freeze becomes visible. TCP has no notion of a host moving; it sees a connection whose segments stop arriving for the duration of the freeze. What happens next depends on how long the gap is and what was in flight.

RFC 9293 gives TCP a retransmission timer that starts at a value derived from the measured round-trip time and backs off exponentially on each timeout. A freeze shorter than the current RTO produces no visible effect at all: the sender’s segments arrive late, the receiver’s ACKs arrive late, and the connection continues. A freeze longer than the RTO causes the sender to retransmit, which is wasteful but harmless; a freeze long enough to trigger several backoffs can push the effective retransmission interval into seconds, so the connection continues but with degraded throughput for a while.

The dangerous case is the one where the source host keeps its network identity. If the source’s virtual switch still has the guest’s tap device attached and still responds to traffic for the guest’s MAC — or if the source’s ARP stack answers for the guest’s IP — then a peer’s retransmissions may reach the source, which has no guest to deliver them to, and the peer may receive a TCP reset from a stack that considers the connection unknown. A reset terminates the connection permanently. Preventing this is a matter of tearing down the source’s network attachment at the right moment: after the destination is confirmed live, before the source’s stack can answer for the address.

The mirror-image problem exists on the destination: if the destination’s switch port has not been reconfigured, frames for the guest may be delivered to a port that has no guest, and a switch that responds to unknown unicast by flooding sends them everywhere, including back to the source. Flooding is usually benign — the extra copies are dropped — but it can produce duplicate segments if the source still has a path to the guest, and duplicate segments are handled by TCP’s sequence checks only if they arrive after the original, not before.

Split-brain and the duplicate-delivery window

The transition has an interval during which both hosts can, in principle, transmit as the guest. If both do, peers see two sources with the same MAC and IP, switches see the MAC flapping between ports, and TCP connections see segments arriving out of order from two paths.

The standard discipline is a strict handover: the source’s network path is disabled before or at the same instant the destination’s is enabled, with the freeze in between ensuring no guest-generated traffic exists during the switch. The order matters:

1. Freeze the vCPUs. No new guest traffic is generated.
2. Drain the source’s transmit queues so that frames already handed to the device are either sent or discarded deterministically.
3. Copy the residual memory and device state.
4. Bring up the destination’s device and network attachment.
5. Send the gratuitous announcement from the destination.
6. Resume the vCPUs.
7. Tear down the source’s attachment.

Steps 4 and 5 happen before 6 so that the switch has re-learned by the time the guest’s first frame is sent. Step 7 happens after 6 so that if the guest’s first frames are lost, retransmission has a working path. The window in which both attachments exist is therefore the window between 4 and 7, and it is kept short but nonzero; during it, only the destination transmits, because the source’s guest is not running.

The sequence is not safe against a source that fails to complete step 7. A hypervisor that crashes after step 4 leaves both attachments configured, and the switch’s forwarding entry will flap between them as the source’s stack occasionally emits frames for the guest. Nothing in the protocol detects this; the operational answer is a fencing mechanism that disables the source’s attachment when the destination reports success, and a timeout on the source that tears down its own attachment if it does not hear from the destination.

What the guest can see

From inside the guest, a migration should look like a brief pause. Its interface counters do not reset, its MAC does not change, its IP does not change, and its TCP connections stay established. The visible artefacts are the ones already described: a burst of retransmissions if the freeze exceeded the RTO, a possible burst of timer expiries, and a momentary gap in received frames.

Guests that send periodic keepalives or run their own failure detection notice the gap if it exceeds their threshold. A guest running a cluster membership protocol with a sub-second timeout will declare itself partitioned if the freeze is longer than that timeout, which can trigger a failover that is far more disruptive than the migration. This is a case where the correct downtime target is set by the guest’s own timers rather than by any host-side constraint, and it is one of the reasons migrations of tightly coupled clusters are treated as planned outages rather than transparent moves.

The device side of the network path — the virtqueue indices that must be restored so the device does not re-deliver frames the guest already consumed — is covered in Serializing Device State Across a Migration. The network layer’s job is to make sure that once those indices are restored, the frames behind them can actually reach the peer.