Staged Rollout Cohorts and What Halt Actually Does

A staged rollout is often described as releasing to a percentage of users. That description is wrong in a way that matters for reasoning about failure: the store does not select users, it selects installs, and it selects them by a hash rather than by a random draw at request time. The distinction determines what happens when the percentage is raised, lowered, or the release is halted.

Cohorts are hashed, not sampled

When a release is set to 10%, the storefront must decide, for each device that checks for updates, whether that device is in the 10%. If the decision were made randomly per request, a device could be in the cohort on one check and out of it on the next, and the rollout’s measured crash rate would mix devices that received the build with devices that did not. The decision is instead deterministic: a stable device or account identifier is hashed, and the cohort is the set whose hash falls below the threshold.

Determinism buys three properties. A device that is in the cohort stays in it across checks, so the update actually arrives. Raising the percentage from 10% to 20% is a superset operation — nobody leaves the cohort — so the devices that already updated are not stranded in a group the store no longer recognises. And the cohort composition is reproducible for analysis, because the hash function and the identifier are fixed.

The threshold comparison uses the low bits of the hash rather than a modulo of a counter, because a counter would require the store to maintain per-device state and would be sensitive to the order in which devices first appear. Taking, for example, the low 32 bits of a hash and comparing against 0xFFFFFFFF * percentage / 100 gives an exact expectation with no state.

The identifier choice is not neutral. Hashing by account means a user with two devices is in the same cohort on both, which is good for measuring user-level outcomes and bad for measuring device-level coverage. Hashing by install means a reinstall can move a device between cohorts. Hashing by device means the cohort survives an account change. Whichever is chosen, the analysis has to use the same unit, or the crash rate denominator will not match the numerator.

Halt is a gate on new adoption, not a recall

The single most consequential fact about staged rollout controls is that halting does not remove the build from devices that already have it. Once an install has completed, the device holds a valid signed APK at a version code the installer accepted. The store can stop offering the update, and it can offer a fixed build with a higher version code, but it cannot reach in and revert.

That asymmetry defines the recovery plan. Halting at 10% caps the affected population at roughly 10% of the install base, which is the entire purpose of the first stage. The follow-up is a new release, not a rollback: the defective build is superseded by one with a higher version code, and devices that took the defective build update again. Devices that never took it skip directly to the fix, which is why version code monotonicity is a prerequisite for the whole strategy rather than a bookkeeping detail.

There is a narrow exception on iOS: a version can be paused for automatic updates, and in some cases a developer can request that a previously released version no longer be offered. Neither of those removes an installed copy. The store’s leverage is always over the offer, never over the installation.

What a percentage actually covers

A 10% rollout does not mean 10% of traffic to the app’s backend comes from the new build. Adoption lags the offer: devices check for updates on their own schedule, some users have automatic updates disabled, and some devices are offline for long periods. The realised share of the new version after a stage has been at 10% for a day is materially below 10%, and the curve approaches the offer percentage asymptotically rather than stepping to it.

This lag is a feature for the first stage and a hazard for the last. As a stage approaches 100%, the remaining population is disproportionately composed of devices that do not auto-update, which correlates with older OS versions, unusual configurations, and users who have deliberately pinned a version. The last percent of a rollout therefore has a different failure profile than the first, and a crash-rate threshold calibrated on the early cohort can produce false confidence about the tail.

The other coverage gap is the split between new installs and updates. A new install of an app during a staged rollout generally receives the current staged version rather than the last full release, so the rollout applies to installs as well as updates. An app whose first-run path is broken will show that failure in the staged cohort even though no existing user updated.

Metrics that can be attributed

The purpose of staging is to compare a cohort against a baseline with enough precision to detect a regression before the whole population sees it. That comparison needs a denominator that is not contaminated. Crash-free-session rate, computed over sessions from the new version, has a denominator that only exists once the version is installed, so it measures the new build but not the difference from the old one. Crash-free-user rate computed over all users in the cohort, regardless of which version they run, measures the cohort’s experience but dilutes the signal as adoption rises.

The workable framing is a ratio with the cohort as the unit: for devices offered the build, compare the rate of the outcome against devices in the control set — those outside the cohort but otherwise comparable. The comparability is the hard part, because cohort membership is a hash of an identifier and is not balanced on OS version, device model, or geography unless the hash happens to be. Any analysis has to check the balance rather than assume it.

Early-life crash rate is the metric that responds fastest, because crashes in the first session after an update do not wait for adoption to accumulate. Metrics that depend on user behaviour over days respond too slowly to gate a rollout that advances on a daily cadence, so the gate is usually a short-window crash threshold plus a manual review of the anomaly feed.

Advancing, pausing, and the cost of each stage

Each stage boundary is a decision point, and the cost of the decision rises with the percentage. Advancing from 1% to 5% risks 4% more installs; advancing from 50% to 100% risks the remainder. A schedule that advances on a fixed cadence regardless of evidence spends the cheap stages without using them.

Two structural limits constrain the schedule. The first is adoption lag: a stage must last long enough for the offer to be realised on a meaningful share of the cohort, or the evidence at the next boundary describes a population that mostly is not running the new build. The second is the response time of the fix: if a defect is found at 10%, the time to build, sign, review, and publish a replacement is measured in days, and during that time the 10% is frozen at the defective build. The stage’s duration should therefore be bounded below by the evidence window and above by tolerance for a frozen population.

Review latency is part of that budget and is easy to forget. A staged rollout that depends on shipping a fix within a day assumes the replacement clears review within a day. Where review is slower, the first stage must be smaller, because the population exposed to the defect will stay exposed for longer. That is the trade the percentage encodes: exposure multiplied by the time to remedy.

Where the model does not apply

Staged rollout assumes the update is optional and the previous version remains functional. It does not apply to a mandatory migration enforced by the backend, where an API version is retired and old clients stop working; in that case the staged rollout offers no safety, because the failure mode is the old version rather than the new one.

It also assumes that the cohort is large enough to detect the defect. A regression that affects one device model at a fraction of a percent will not appear at 1% regardless of the metric, and will appear only after the rollout has covered enough of that model. For those, the practical control is not the percentage but the pre-release channel — a beta track whose population includes the affected configuration, whose participants receive builds through the same delta update mechanics as production users and therefore exercise the same install path. The staged percentage and the beta population answer different questions, and neither substitutes for the other.