Rolling Back a Change That Already Mutated State

Reverting a deployment is a single command in most systems, and it restores the previous binary within the time it takes to replace the instances. What it does not do is restore the data. Every write the new version performed remains, every message it published remains in the queue, every external system it called has already recorded the call. The gap between “the code is the old code again” and “the system is as it was” is where rollback plans fail, and it is a gap that has to be closed by design before the deployment starts.

The inverse of a write is not a delete

Consider a change that adds a discount to orders: the new build computes a discounted total and writes it into a total column that previously held the undiscounted amount. Reverting the binary stops new discounts from being written. It does not recompute the totals of the orders processed during the rollout, and those orders are now wrong in a way the old code cannot detect, because the old code has no concept of a discount and reads total as authoritative.

The naive remedy, whether deleting the affected rows or deleting the column, fails for reasons that are specific to each case. Deleting rows loses the customer’s order. Deleting the column loses the pre-change value, which was overwritten in place. Neither is an inverse, because the original write was destructive: it replaced a value rather than appending information.

The rule that follows is that a mutation is reversible only if it preserves enough information to reconstruct the prior state. Appending a row, adding a column, publishing a message, or writing a new version of a record under a new key all preserve the old state, and the inverse is to stop reading the new data or to mark it as not-applied. Overwriting a value in place destroys the old state, and no amount of code reversion brings it back.

Compensating actions, not inverses

Where the state cannot be restored, the available technique is a compensating action: a new operation that produces a state equivalent to the one that would have existed had the original operation not occurred, without requiring the original operation to be undone. This is the mechanism Sagas formalise for long-lived transactions, and its essential property is that the compensation is itself a forward operation with its own failure modes, not a rollback.

The distinction is concrete. If the discount change added 10 percent off, the compensation is not “restore the old total”, because that value is gone, but “credit the difference back” or “reissue the order at the correct total”. Both are new writes, both are visible to downstream consumers, both can fail, and both must themselves be idempotent so that a retry of the compensation does not double-credit. The system moves forward into a state that is acceptable, rather than backwards into a state that no longer exists.

-- Compensating action: append a correcting entry rather than overwrite.
INSERT INTO ledger (order_id, kind, amount, reason)
SELECT o.id, 'adjustment', o.total - o.undiscounted_total, 'rollback:discount-v1'
FROM orders o
WHERE o.applied_version = 'discount-v1'
  AND o.total <> o.undiscounted_total
  AND NOT EXISTS (
    SELECT 1 FROM ledger l
    WHERE l.order_id = o.id AND l.reason = 'rollback:discount-v1'
  );

The NOT EXISTS clause is what makes the compensation re-runnable. Without it, a retry after a partial failure credits the adjustment twice, and the second credit is indistinguishable from a legitimate second adjustment. Every compensating action needs an equivalent guard, and the guard has to be enforced by the store rather than by the caller, because the caller is the component that just crashed.

Classifying a change before it ships

The useful question to answer at design time is not “can this be rolled back” but “what does the previous version do with data written by this version”. Three answers are possible, and they map onto three deployment shapes.

The previous version reads it correctly. The change is transparent to the old code: a new nullable column it ignores, a new field in a message it discards, a new index it does not use. Rollback is a redeploy, and the deployment can be a single release. This is the cheapest case and the one to aim for; it is achieved by making the new representation additive.

The previous version misreads it, but the effect is recoverable. The old code treats an unknown value as a default, or reads a new column as null and takes a fallback path that is wrong but not destructive. Rollback requires a data step in addition to the redeploy: reverting the affected rows, recomputing derived values, or replaying from a point before the change. The deployment must therefore include a plan for that step, and the step must be tested, because it runs under pressure.

The previous version cannot function with it. A dropped column, a changed encoding, a message format the old consumer cannot parse. There is no rollback; the only path is forward. Changes in this class should either be split so that the irreversible step is separable and last, or shipped with an explicit acknowledgement that the deployment is one-way and with a forward-fix plan instead of a revert plan.

The classification is cheap to perform and expensive to omit, and it is the same question that decides whether a change needs the three-phase treatment rather than a single release. It takes one question per changed interface and it determines whether the deployment configuration should permit a revert at all. A system that offers a rollback button for a one-way change is offering a button that will corrupt data when pressed.

The window between decision and effect

Even a reversible change has an interval during which both the mutation and its absence are live. The rollout is aborted at time T, the last instance of the new version stops serving at T + d, and during d the new version continues to write. Any compensation computed from a snapshot taken at T will miss the writes in that interval, and the compensation will be incomplete.

Two approaches handle it. The first is to make the mutation self-identifying: every row written by the new version carries a marker such as a version column, a flag, or a distinct key prefix, so the compensation can select by marker rather than by time window. This survives the interval, because the selection criterion is a property of the write rather than of when it happened. The second is to stop the mutation before aborting: disable the new behaviour through a configuration switch that both versions read, wait for the effect to propagate, and only then replace instances. The second is faster to execute and requires a switch that exists and is honoured by the new build, which is a design commitment made well before the deployment.

The version-marker approach is the more reliable of the two, because it does not depend on a control plane that may itself be unhealthy during the incident. Its cost is a column or a field on every affected record and a discipline of writing it consistently, which is small compared to the cost of reconstructing which rows were touched.

Reverting derived state

Derived state is the hardest case, because it can be regenerated in principle and cannot be regenerated cheaply in practice. A cache populated by the new version, an index built with a changed analyser, a materialised aggregate computed under new rules. Each is a function of primary data plus code, so reverting the code and discarding the derived state restores correctness. The cost is the rebuild.

That cost sets a bound on how much derived state a deployment may touch. Rebuilding a cache is a warm-up measured in minutes; rebuilding a search index over a large corpus is measured in hours; recomputing a warehouse aggregate may be measured in days. A change that alters the derivation rule for the last category cannot be rolled back within the incident’s time budget, which means the rule change needs the expand-migrate-contract treatment: compute the new derivation into a new location, compare it against the old, and switch readers only once the new location is complete. The comparison step is the one that establishes the new derivation is correct before it becomes the only one, and it is the same mechanism as verifying a backfill by range digest applied to a derived dataset.

Rehearsing the reverse path

A rollback plan that has never been executed is a hypothesis. The properties it depends on are all falsifiable: that the old version can read the new data, that the compensation selects the right rows, and that the rebuild completes within the window, and the place to falsify them is a staging environment with data written by the new version, not an incident.

The minimum rehearsal is the mixed state: one instance of the old version running against a store that the new version has written to, exercised through the paths the old version uses. That test catches the misread cases and the schema incompatibilities. It does not catch the compensation’s correctness, which requires running the compensation against a known input and comparing the result, and it does not catch the rebuild’s duration, which requires measuring the rebuild on data of realistic size.

The last of those is the one most often skipped and most often decisive. A rollback that takes forty minutes to restore service is not a rollback; it is an outage with a plan attached. Knowing the duration in advance is what allows the decision to revert to be made early, while the alternative of rolling forward with a fix is still cheaper than the reverse path. That comparison, between the cost of going back and the cost of going forward, is the actual content of the rollback decision, and it can only be made with both numbers known.