Cloud Infrastructure · Cloud Migration

The Cutover Boundary of a Workload and Why It Is Never One Process

How to decide which processes, data stores, and identities must move as one unit, what breaks when a boundary is drawn too small, and the cost of drawing it too large.

Every migration plan contains an implicit decision that is rarely written down: which parts of the system move together. The decision is usually made by looking at a repository, a deployment unit, or an organisational chart, and all three answer a different question than the one being asked. A migration boundary is a consistency boundary. It is the smallest set of components that must change state atomically, or at least in an order that no external observer can detect, in order for the system to remain correct across the move.

A boundary is defined by what must stay consistent

Consider a service that writes to a database and publishes an event to a queue after each write. Move the service and leave the database, and for the duration of the cutover, writes go to the old database while events flow from the new environment, or the reverse. The window in which the two disagree is not a performance problem; it is a correctness problem, and it is bounded below by however long it takes to redirect traffic and drain in-flight requests.

The same reasoning applies to anything with a durable side effect. Three categories recur:

  • Transactional state. A relational database, a document store, or a queue with a delivery guarantee. Two writers on the same data with no coordination protocol produce divergence that no reconciliation can distinguish from legitimate concurrent updates.
  • Identity and credentials. A service authenticates with a key or certificate that is valid in one environment and not the other. Moving the service without the identity splits the boundary, and the failure appears at the first token expiry rather than at cutover.
  • Address and naming. The service’s own name and address, and the names of anything it calls. If a caller resolves a name that changes meaning at cutover, the caller is inside the boundary whether or not the plan says so.

The boundary, then, is the transitive closure of these three relations starting from the component being moved. Computing it is mechanical, and it is the same traversal as the observed dependency graph, with edges filtered to those that carry one of the three properties. Edges that carry only best-effort traffic, such as metrics shipping or a cache that tolerates a miss, can be cut without entering the closure.

Drawing the boundary too small

The characteristic failure of a too-small boundary is a split-brain window in which both halves are functioning and disagreeing. It is insidious because every individual component reports healthy. Nothing throws an error; the two databases simply accept writes that the other never sees, and the divergence is discovered later, usually by a report that does not add up.

The narrower version of the same failure is ordering. Suppose the plan moves a consumer before the queue it consumes. For the interval between the two moves, the consumer connects to a queue endpoint that has moved or not moved, depending on which half of the name resolution it sees. Retry logic hides this. The consumer retries, succeeds after the queue moves, and the plan records the step as clean. What the retries did was reorder the messages relative to each other, and any consumer that assumed per-key ordering now has a bug that will surface under load weeks later.

A third failure mode is invisible to the migration team entirely: the boundary excludes a party that is not in the inventory. A partner integration, a scheduled job on a machine nobody owns, or a report generated by an analyst’s local script all consume the system from outside the observed graph. These appear as unexplained traffic in the destination after cutover, which is a benign outcome, or as silence from a caller that nobody notices is silent.

Drawing the boundary too large

The opposite error is to enclose everything reachable, which is what a plan does when its authors are uncertain. The cost is not conceptual; it is scheduling. A boundary of one component can be moved in an afternoon and reverted the same day. A boundary of thirty components has a cutover window measured in hours, requires every owning team to be present simultaneously, and has a rollback procedure whose own correctness must be established before it can be relied on.

Large boundaries also amplify the blast radius of a bad decision. If the move is wrong for one reason, all thirty components revert together, and the organisation pays the full cost of the attempt rather than a fraction of it. The relationship is not linear: coordination cost grows with the number of interfaces inside the boundary, which is roughly quadratic in the count of components for a densely connected system.

There is a real benefit to a large boundary, which is why the temptation exists. Inside the boundary, no protocol is needed between old and new. No dual-write, no change stream, no reconciliation. The system stops and starts. That simplicity is worth paying for when the internal interfaces are numerous and poorly understood, and it is exactly the wrong trade when they are few and well documented.

The two cutover shapes and what each requires

Once the boundary is fixed, the move takes one of two shapes, and they demand different machinery.

Stop-and-copy takes the boundary offline, copies its state, and starts it in the destination. The requirement is a quiescent source: every writer stopped, every in-flight transaction committed or aborted, every buffer flushed to durable storage. The copy can then be a plain byte copy or a logical dump. Downtime equals copy time plus verification time plus the time to redirect clients, and copy time grows with data volume at whatever the link and the storage allow. Stop-and-copy is correct by construction and is the right choice whenever the downtime is acceptable.

Copy-and-catch-up copies a snapshot of the state while the source keeps serving, then applies the changes that happened during the copy, then switches. This requires the source to expose an ordered change stream with a known starting point: a write-ahead log with a log sequence number, a binlog with a position, a change-data-capture feed with an offset. The correctness requirement is that the snapshot corresponds to a defined position in that stream. A snapshot taken without recording its position cannot be replayed, because the replay has no way to know which changes are already included. This is the mechanism examined in bulk copy and change stream, and it is the only shape available when the downtime budget is near zero.

The choice between them is not a matter of sophistication. Stop-and-copy has fewer moving parts and fewer ways to be subtly wrong. Copy-and-catch-up trades that safety for availability and pays in machinery that must be tested against the source’s actual behaviour, including the behaviour of long-running transactions, which hold a position in the stream open and prevent the catch-up from ever completing.

Identities and names are part of the boundary, not an afterthought

A cutover changes addresses. The boundary must therefore include every party that holds a cached or configured copy of an address, or the plan must account for how those caches expire. A client that resolved a name once at startup and never re-resolves will keep sending to the old address indefinitely, and no amount of correct server-side work will reach it.

The practical consequence is that the boundary usually extends one hop beyond the last server, into the client population, in the form of a name whose indirection allows the server’s address to change. That indirection is the reason DNS TTLs govern the cutover clock, and the reason a boundary drawn purely from server-side components is incomplete. The same holds for credentials: a certificate or key that the destination environment will not accept must be replaced before the move, not after, and the replacement must be validated against every caller, since a caller that pins a certificate by fingerprint will reject the new one even though the name is unchanged.

What the boundary decision should leave behind

The output of this analysis is not a diagram. It is a list of components with a flag for each: inside the boundary, outside but consumed, outside and consuming, or unclassified. The unclassified set is the one that determines whether the plan is ready, because every entry there is a party whose behaviour at cutover is unknown.

The list also fixes the unit of rollback. If the boundary is correct, reverting means restoring the source to the state it had at the switch and pointing clients back, and the destination’s state is discarded. If the boundary was drawn too small, reverting is not possible without reconciling two divergent states first, which is a different and much harder operation. That asymmetry is why rollback is a question about write reversibility rather than about how quickly a switch can be flipped, and why the boundary deserves to be settled before any data moves rather than while it is moving.

References

  1. RFC 5280: Internet X.509 Public Key Infrastructure Certificate and CRL Profile — IETF
  2. Site Reliability Engineering: How Google Runs Production Systems — O'Reilly Media, Beyer et al.
  3. The Twelve-Factor App — Adam Wiggins