Rollback as a Question of Write Reversibility

A rollback plan written as “repoint DNS to the old environment” answers a routing question. The question that decides whether a rollback is possible at all is different, and it is about state: after the destination has accepted writes for an hour, the source has never seen some of what exists there, and reverting means moving it back or deciding to lose it.

Two different reversions

Traffic reversion and state reversion are independent operations with different costs. Traffic reversion is a control-plane change and completes as fast as the indirection propagates. State reversion is a data operation whose cost scales with the number of writes the destination accepted and with how much of the source’s own history those writes displaced.

If the destination is read-only during the trial period, state reversion is free: there is nothing to move back, and reverting traffic restores the previous state exactly. If the destination accepted writes, state reversion requires one of the mechanisms below, and each has a failure mode that must be established before the cutover, not after.

The asymmetry is the point. A migration that is reversible is one where the writes accepted at the destination are either forbidden, logged in a form the source can consume, or applicable to both. A migration that is not reversible is one where the destination is the only copy of some data and no path leads back.

Why replaying the log backwards is not a rollback

The intuition that a write-ahead log can be undone by replaying it in reverse is wrong in a way that matters. A log records the physical or logical changes that were applied. Reversing a change requires knowing the prior value, which a logical log of UPDATE t SET v = 5 WHERE id = 7 does not contain. It says what the value became, not what it was.

Systems that support true undo record before-images. Database rollback of an uncommitted transaction works because the undo information is kept for exactly that purpose, for exactly the transactions that are still open. Once committed, the before-image is gone; the log is for recovery forward, not for reversal.

Consequences follow directly. If the destination applied a write that changed a value from A to B, and the source still holds A, then reverting means discarding the destination’s B and continuing from A. That is only correct if B has no effects outside the destination: no notification was sent, no downstream system consumed B, no report was generated. Any of those makes the revert a divergence rather than a restoration.

Dual-write and its ordering problem

The standard mitigation is to write to both stores during the trial period, so that either can be authoritative at the moment of reversion. The mechanism is straightforward in the happy path and has a well-known failure mode in the unhappy one.

Writing to two stores without a distributed transaction means the two writes can partially fail. If the source write succeeds and the destination write fails, the two diverge immediately. If the order is reversed, the destination holds a record the source never received. Neither ordering is safe, and choosing the order only decides which of the two divergences occurs.

The usual mitigation is to make the second write asynchronous and retryable: write to the authoritative store synchronously, enqueue the second write, and reconcile the queue. This gives up atomicity and gains a bounded divergence window that a reconciliation process can close. It requires that the second write be idempotent, which means it must be expressible as a set operation rather than an increment, or it must carry a unique identifier that the second store uses to discard duplicates.

A second problem is ordering across writes. Two writes that arrive at the destination out of order produce a state that neither ordering would have produced on its own. Per-key ordering is usually enough and requires the queue to partition by key; global ordering is expensive and rarely necessary. Where the destination applies writes concurrently, per-key ordering must be enforced at the destination rather than at the queue.

The cost of dual-write is that it doubles write load and adds a failure path that must be tested. Its benefit is that rollback remains available for the duration, which is exactly the property that makes a staged migration worth doing. The alternative, a one-way cutover with a backup as the fallback, is simpler and makes the fallback a restore operation whose recovery point is the last backup rather than the last moment.

Tombstones, soft deletes, and the delete problem

Deletes are the operation that most often makes a cutover irreversible. A row removed at the destination during the trial period, then reverted, must be restored at the source. If the source’s row still exists, nothing needs to happen; if the delete propagated to the source, the row is gone and must be recovered from a backup, which loses every subsequent modification to that row.

Two designs address this. The first is a soft delete: the row is marked rather than removed, and the mark is reversible by clearing a flag. The cost is that every reader must filter on the flag, which is a permanent change to every query and an easy one to forget. The second is a tombstone in the change stream: the delete is recorded as an event with the key and a timestamp, and the event is retained long enough to cover the rollback window. Restoring then means reinserting the row from a retained copy.

The retention period is the rollback budget expressed in time. A tombstone retained for seven days means a rollback is possible for seven days, provided the row’s pre-delete content was also retained. Retaining the content is a second decision, and it is the one that gets skipped, because tombstones are small and before-images are not.

What the rollback budget actually contains

The set of properties that must hold for a rollback to be possible, at a given moment after the switch, is finite and can be enumerated:

The source must still exist and must be able to serve. This sounds trivial and is the most common failure. A source that was scaled down to zero, or whose licence was not renewed, or whose credentials were rotated at the switch, cannot serve regardless of what the data says.

The source must be able to accept the writes that happened at the destination. That requires those writes to be either forbidden, dual-written, or captured in a stream the source can apply. Capturing them in a stream is the strongest option and requires the destination to expose its own change feed, which not every store does.

The source must be able to reject writes that arrived at the destination and were subsequently reverted elsewhere. This is the case where a downstream consumer acted on a destination write and cannot un-act. Retaining the consumer’s ability to compensate is part of the budget, and it usually lives outside the migration plan.

Each property has an expiry. The source’s ability to serve expires when it is decommissioned. The ability to apply captured writes expires when the captured stream is discarded. The ability to compensate expires when the downstream system’s own retention lapses. The rollback window is the minimum of these expiries, and it should be written into the plan as a date rather than as a procedure.

Deciding that the cutover is one-way

Some migrations are honestly irreversible, and saying so is better than maintaining a rollback plan that cannot be executed. Three conditions make a cutover one-way:

The destination’s schema cannot represent the source’s state, so a revert would require a lossy transformation. A migration that normalises a denormalised structure falls here; the information needed to reconstruct the original is gone.

The destination’s writes have external effects that cannot be compensated. Money moved, goods shipped, notifications delivered. Compensation is possible in principle for many of these and requires a compensating transaction for each, which is a larger project than the migration.

The source cannot be kept operational for the duration. A vendor contract that ends, hardware that is being returned, or a licence that expires at a fixed date all convert the migration into a deadline rather than a choice.

Recognising one-way status changes the plan rather than ending it. A one-way cutover justifies more rehearsal, because the fallback is a restore rather than a revert, and the restore’s recovery point is the last verified backup rather than the last write. It also justifies a longer dual-run period, because the destination must be proven correct before the switch rather than corrected after it.

The distinction also clarifies what evidence is worth collecting. For a reversible cutover, the evidence that matters is that reversion works, which is testable by rehearsing it. For a one-way cutover, the evidence that matters is that the destination produces correct results for the full range of inputs, which is what shadow comparison and reconciliation after the switch are for. Collecting the wrong kind of evidence is how a plan ends up with a rollback procedure that has never been executed and a destination that has never been compared.