Rollback Without Downgrade: Containment After a Bad Release Ships

The word rollback implies a mechanism that mobile distribution does not have. Once a package is installed, the installer’s only forward move is a higher version code signed by the same identity; there is no operation that restores the previous package, and even a successful downgrade would leave the new version’s data on disk in whatever state it reached. Every rollback strategy for mobile is therefore a form of containment: changing what the already-installed code does, rather than changing which code is installed.

The four levers, ordered by reach

Containment mechanisms differ in how much of the app they can affect and how quickly.

Store-side halt stops offering the build. It affects devices that have not yet updated and nothing else. Its latency is one store refresh cycle.

A superseding release ships a new version with a higher code. It reaches devices on the update schedule, which for a staged rollout means a fraction per day, and it requires the whole build-verify-publish path — the slowest of the levers.

Remote configuration changes values the app reads at run time: which endpoint it talks to, which feature is enabled, what a threshold is. It reaches every device that makes a configuration request, which is typically the next app launch, and it requires no signing and no review.

Feature gating is the app-level version of the same idea: the defective code path exists in the binary but is not reachable because a gate is closed. Its reach is identical to remote configuration, and its advantage is granularity — it can disable one code path without changing a shared configuration value that other behaviour depends on.

The levers compose. A halt caps exposure, a gate disables the defect inside the exposed population, and a superseding release eventually removes the defective code. The mistake is to treat any one of them as a rollback.

Why the app must be built to be contained

A remote configuration mechanism cannot disable code that was not written to consult it. This sounds obvious, and it is the single constraint that makes containment a design property rather than an operational one. Retrofitting a gate onto a defect discovered after a release ships requires shipping a new binary — which is the lever that was too slow in the first place.

The design rule that follows is: any behaviour whose failure would be severe enough to require immediate containment must be reachable through a gate that can be closed without a release. That is not the same as gating everything. Gating every code path multiplies the number of states the app can be in, and each gate is a branch that must be tested in both positions. The useful discipline is to gate the transitions: the point where the app starts using a new code path, new data format, or new backend endpoint, rather than every function inside it.

A gate at a transition has a further property that a gate inside a feature does not: it can be closed before the transition begins, which means the new path is never entered and the old one continues. A gate inside a feature leaves the app in a half-migrated state when closed, and the old path may no longer be present.

// The gate guards entry into the new path, not the work inside it.
enum Renderer {
    case legacy, next

    static func select(config: RemoteConfig, installed: Version) -> Renderer {
        guard config.nextRendererEnabled else { return .legacy }
        guard installed >= config.nextRendererMinVersion else { return .legacy }
        return .next
    }
}

The version floor in the second condition is what makes a gate usable across a rollout. The gate can be opened for builds at or above a version that has been validated, while older installed builds continue on the legacy path, so a single configuration change addresses a heterogeneous installed population.

Data is the part that does not come back

Code containment is reversible. Data migration is not. If a release rewrote a local database into a new schema, closing a gate stops further migration but does not restore the rows already converted, and the old code path cannot read the new format.

Three properties make a migration containable, and all three have to be designed before the release ships.

Forward compatibility. The old code must be able to read data written by the new code, at least for the subset of fields it uses. If the new schema adds columns, the old reader ignores them; if it renames or re-encodes a field, the old reader breaks. Keeping additions additive and leaving the old encoding in place during the transition is the standard discipline, and it costs storage and a second write path.

Reversibility of the marker. The schema version marker discussed in version code monotonicity is a single integer. A migration that has run to completion cannot be undone by resetting it, because the data has already changed shape. The marker only supports rollback when each step is individually reversible, which in practice means the migration must keep the pre-migration representation until the step is confirmed.

Dual-write during the window. Writing both representations while the gate is open lets either reader function, at the cost of doubled write volume and a reconciliation path for the period when both exist. It is the only mechanism that makes a schema change survivable under containment, and it is expensive enough that it is reserved for changes where the alternative is unacceptable.

What a kill switch can and cannot reach

The failures that containment handles well are behavioural and server-mediated: an endpoint returning bad data, a feature with a logic defect, a threshold that is wrong for some population. In each case the app consults a value that can be changed, and the change takes effect at the next consultation.

The failures it does not handle are those that occur before the app can consult anything. A crash during startup, a crash in the code that fetches configuration, a defect in the migration that runs before the first configuration read, and a native crash in a library that loads before the app’s own code — none of these can be gated, because the gate is downstream of the failure. The proportion of severe mobile defects that fall into this category is large enough that containment cannot be the only strategy; it is why staged rollout exists as the primary control and containment as the secondary one.

There is also a failure mode in the containment mechanism itself. A configuration service that becomes unavailable must not block the app, so the app falls back to defaults — and the defaults are whatever was compiled in, which may be the very behaviour that was gated off. Whether the fallback is “feature on” or “feature off” is a decision that has to be made per gate and encoded at build time. A gate whose default is “on” and whose server is unreachable is a gate that has been silently reopened.

Sequencing a containment event

The operational sequence for a defect found during a staged rollout is bounded by the levers’ latencies, and the order matters more than the individual actions.

First, stop the exposure: halt the rollout. This is a single action with an immediate effect on new offers, and it costs nothing to do early. Second, decide whether containment is possible — whether the defect is behind a gate, and whether the gate’s closure leaves the app in a coherent state. Third, if containment is possible, close the gate and confirm through the crash or error signal that the closure took effect, which requires the signal to be segmented by gate state rather than only by version. Fourth, if containment is not possible, the remedy is a superseding release, and the rollout stays halted until it is available.

Two details are easy to get wrong. The signal has to distinguish devices that received the configuration from those that did not, or the effect of the gate is averaged away and looks like no effect. And the superseding release must be built from the last good source, not from the defective branch with a fix applied on top — reverting the change is faster to validate than repairing it, and the repair can ship afterwards.

The cost of keeping containment available

Every gate is a state that must be tested and eventually removed. A gate that stays open for a year is a branch nobody exercises, and the code behind it has drifted from the code in front of it. Removal is a release of its own, and the discipline that makes it happen is to attach an expiry to each gate at the time it is added and to treat an expired gate as a defect.

The alternative — no gates, no containment, rely on fast releases — is coherent for an app whose release latency is measured in hours and whose installed population updates quickly. For an app where a significant share of the installed base runs a version several releases old, containment is the only mechanism with reach into that population, because it does not require the device to update at all. That asymmetry, rather than any property of the gates themselves, is what justifies their cost: they are the only lever that works on devices that will never take the next release.