Rollback Boundaries: Which Layers a Reversal Can Restore

Rollback is usually described as a single operation: restore the previous version and the incident is over. In practice a deployment touches several layers that reverse at different costs, and the layers do not reverse together. Reverting an image is a routing change measured in seconds. Reverting a schema change is a migration with its own risks. Reverting a backfill that overwrote source values is a restore from backup, and the restore loses every write made since it was taken.

Designing a deployment strategy therefore means deciding, in advance, which steps can be undone and which cannot, and ordering the work so that the irreversible ones come last.

The layers and what each reversal actually costs

Code reverses by serving the previous artefact. The cost depends entirely on whether that artefact is still deployable and still running. If the previous version’s instances were terminated at cutover, rollback means pulling an image, starting instances, warming caches and connection pools, and passing readiness — the same cold-start cost that made the forward deployment nontrivial, now under incident conditions. Keeping a small number of previous-version instances idle, or retaining the previous image with a fixed retention policy, is what converts a slow rollback into a fast one.

Configuration and flags reverse without an artefact. Flipping a flag back is a write to the flag system, effective within its propagation delay; the caveat from feature flags versus traffic routing is that the flag only controls the behaviour behind it, so a defect outside the guarded region is unaffected. Configuration held in the environment or in a mounted file reverses only by redeploying, unless the process re-reads it.

Schema reverses in one direction. Adding a column or an index is reversible by dropping it. Dropping a column is not reversible without a restore, and neither is a type change that discarded information. This asymmetry is the reason for the expand/contract sequence in expand and contract schema migrations: every intermediate state of that sequence is one the previous version can still operate against, so a rollback during the overlap is a code change only.

Data reverses only if the transformation is recomputable. A backfill that filled null values from another column is recomputable; a backfill that overwrote the source column in place is not; a deletion is not, except through a restore. Rows written by the new version while the old version could not represent them are a distinct problem: rollback succeeds mechanically and leaves the old version reading values it cannot interpret.

Messages and events do not reverse. Once a message is published, consumers may have processed it, and there is no general way to un-publish. Systems survive this by making consumers tolerant (ignoring event types they do not recognise, and treating unknown fields as opaque) which is the event-schema analogue of the additivity rule for columns. A consumer that fails hard on an unknown event type turns a code rollback into a queue backlog.

External effects do not reverse at all. An email sent, a payment captured, a record created in a partner’s system: rolling back the code does not unsend it. They require a compensating action, which is a different operation with its own failure modes, and the operations that need one should be identifiable before a rollout begins rather than discovered during one.

The first irreversible write is the point of no return

Each of these layers becomes irreversible at a different moment, and the earliest of those moments is the point after which backward and forward repair are no longer equally available. The useful discipline is to name that moment explicitly for each change and to schedule everything irreversible on the far side of a deliberate pause.

A concrete ordering makes the shape clear. A change that adds a field to a stored entity and starts populating it procedurally:

1. Deploy code that writes both the old and new representation and reads either. Reversible; nothing visible changes.
2. Backfill existing rows. Reversible if the source values are retained; recomputable if the batch is wrong.
3. Verify. Both versions can still run: the old one ignores the new field.
4. Flip the flag that makes the new representation authoritative for reads. Reversible.
5. Observe for a period comparable to the change’s slowest cycle — a billing period, a weekly batch, a certificate rotation.
6. Drop the old representation. Irreversible. This is a separate change, carrying its own review, and it is the only step in the sequence that cannot be walked back by re-running the earlier ones.

Steps 1 through 5 can be reversed by running backwards. Step 6 changes the answer to the question of whether the change can be undone from yes to no, and every hour spent in step 5 is buying certainty about step 6 at the cost of storing both representations.

The ordering rule generalises: irreversible operations should be last, small, and separate. Last, so that the maximum amount of evidence is available before committing. Small, so that the operation’s duration and lock footprint are bounded — a drop and a table rewrite that were combined into one migration cannot be aborted halfway. Separate, so that the decision is a distinct event with a distinct approval rather than a side effect of pushing the application build.

Recovery objectives and what a restore really costs

For layers that can only be repaired from backup, the recovery plan’s numbers apply. A recovery point objective states how much recent data loss is tolerable; a recovery time objective states how long the repair may take. Both are usually recorded for disaster recovery and rarely connected to deployment planning, which is a mistake, because a restore is the only remedy for an irreversible data mistake and its cost is measured in the whole interval since the last usable backup plus everything written after the restore point.

Two properties of a real restore are easy to discount. First, a full restore commonly means losing the writes made after the backup, which must then be reconciled by hand or by replaying a log — feasible for a log-structured system with archived write-ahead log segments, impractical for a system whose only copy is a nightly dump. Second, a restore is not a rollback of the schema alone: it returns the data to a state that the current application may not be able to read, so a restore during an incident usually forces a code rollback as well, and the two must be sequenced.

Keeping a rollback window that is actually open

A rollback window is only real if everything needed to traverse it is still in place. Enumerating those requirements makes the closing conditions visible:

• Artefact availability. The previous images are still in the registry, pinned by digest rather than by a mutable tag, and any dependency artefacts they fetch at start-up are still reachable.
• Capacity. Either the previous fleet is still running, or the cluster has room to schedule it, and the autoscaler has the quota to grow it.
• Configuration. The previous version’s configuration values are recoverable, including the values of any flags, which may have been changed since.
• Credentials. The previous version can still authenticate. A rollout that rotates a key, a certificate, or a database credential can leave the old version unable to start even though its code is intact; the window closes at the rotation, not at the deployment.
• Schema. The previous version remains compatible with the current schema, which is exactly the constraint that expand/contract preserves and that the contract step ends.
• Data. No write has occurred that the previous version cannot represent or interpret.
• Message compatibility. Consumers of the previous version tolerate anything published during the overlap.

The window closes at the first of these that fails, not at the last, and each has a different clock. Schema compatibility ends at the contract migration; credential compatibility ends at the key rotation; artefact availability ends at the retention policy’s expiry. An honest rollback policy states the window as a minimum over those clocks rather than as a single nominal duration, because the binding constraint is usually the one nobody was watching.

Choosing forward over backward

When the rollback window has closed (or when the fix is small and the cause understood) the remaining option is to repair forward: ship a corrective version rather than restore the previous one. The trade is explicit. Forward repair preserves the data changes already applied and avoids a restore; it requires that the current version be the faster thing to change, and that a second failure not compound the first. Backward repair is better when the current version is untrustworthy in ways that are not yet understood, because the previous version is known to work in the conditions that existed before.

The deciding question is usually about evidence rather than preference: how much of the current version’s behaviour is understood, and how long the diagnosis is expected to take relative to the time a rollback would take. Where both are slow (a canary with an underpowered analysis, a low-traffic service, a change whose effects appear only after a full billing cycle) the practical protection is to keep the change small enough and the two representations parallel long enough that neither decision has to be made in a hurry.