Decision Rules for Aborting or Promoting a Rollout

A rollout needs a rule that turns observations into a decision: promote, hold, or abort. The rule is written in a configuration file that looks administrative (a threshold, a window, a number of consecutive bad evaluations) and its behaviour is a statistical procedure whether or not anyone designed it as one. Getting it wrong produces the two failure modes that make teams abandon automation: a gate that aborts healthy rollouts often enough that people route around it, and a gate that watches a genuinely broken version serve traffic for an hour because its threshold was set to a round number.

The arithmetic of the second failure is the easier one to fix, and it starts with the observation that the sample a canary receives is usually much smaller than intuition suggests.

The arithmetic of a small sample

Consider comparing an error rate between two versions. The quantity that governs whether a difference can be resolved is the number of failing requests, not the total. Halving a 1% error rate means distinguishing 1% from 0.5%, and at conventional power (80% at a two-sided 5% level) that requires on the order of five thousand requests per version. Halving a 0.1% error rate means distinguishing 0.1% from 0.05%, which requires roughly ten times as many requests per version — tens of thousands. The scaling is not linear in the error rate but quadratic in the difference between the rates, so each tenfold reduction in the failure rate costs about a tenfold increase in sample.

These figures follow from the standard sample-size formula for comparing two proportions and are arithmetic, not measurement. Their operational consequence is blunt: a canary carrying 2% of the traffic of a service doing twenty requests per second receives roughly twenty-four requests per minute, so it reaches five thousand requests in about three and a half hours, and tens of thousands only over days. If the rollout cadence assumes a fifteen-minute bake, then the gate cannot be a hypothesis test on error rates. It has to be something else: a threshold on a coarser signal, a targeted canary that names the specific workload, or an explicit acceptance that the decision is being made on the basis of a handful of observations and that detection will come from the general alerting rather than from the rollout.

Why fixed-horizon tests break under continuous monitoring

The textbook comparison assumes the sample size is fixed in advance and the test is run once. Rollout controllers do the opposite: they evaluate a condition every interval and abort the moment it is met. Each evaluation is another opportunity to cross a threshold by chance, and the probability of crossing at least once grows with the number of evaluations. For a two-sided 5% test evaluated repeatedly on accumulating data, the actual false-abort probability over a long sequence approaches certainty, not 5%. The phenomenon is optional stopping, and it is the reason a controller with interval: 1m and a 95% confidence threshold will abort healthy rollouts.

Three families of remedy are in use, and they differ in what the operator has to specify.

Group sequential designs divide the rollout into a fixed number of analysis points and spend the total false-positive budget across them, with an alpha-spending function that allows more evidence to accumulate before early stopping produces a decision — an early look costs a stricter threshold, a late look a looser one. The designer chooses the number of looks and the spending shape, and the analysis remains valid at each of those looks. A controller that evaluates continuously does not implement this; one that evaluates at configured milestones can.

Sequential probability ratio tests and always-valid inference take a different route: instead of controlling error at a fixed sample size, they construct a decision boundary (or, in the modern formulation, a confidence sequence) that is valid at every sample size simultaneously. A confidence sequence is wider than a fixed-sample interval at any single point in time, and the extra width is the price of being allowed to stop whenever the evidence crosses. The practical difference is that the operator specifies a minimum effect and a willingness to keep going until a decision is reached, rather than a sample size and a p-value threshold.

Burn-rate alerting abandons significance testing in favour of an explicit budget. The service’s error budget is the complement of its objective over a window; a burn rate of one consumes the budget exactly at the intended pace, and thresholds such as “burn rate above 14.4 sustained for an hour” are chosen so that a certain fraction of the budget is consumed before a page fires. Applied to a rollout, the same idea gives a gate that fires when the new version’s contribution to budget consumption exceeds what the objective can afford, which is a threshold with an operational meaning rather than a statistical one.

The choice between them is largely about who has to defend the number. A burn-rate threshold can be explained to anyone who knows the service level objective. A p-value threshold invites arguments about test design that the rollout rarely deserves.

Abort criteria that survive contact with operations

A decision rule is more than a threshold. Four properties determine whether it is usable.

Direction and magnitude per signal. A rising 5xx rate is unambiguously bad; a rising 4xx rate is ambiguous, and a decline in request volume may be the healthiest possible response to shedding overload. A rule that collapses all signals into one score loses the ability to express this and produces aborts that operators overrule, which then trains everyone to overrule aborts.

A minimum sample before any evaluation counts. Evaluating a ratio on twelve requests produces wild estimates, and a rule with no minimum will abort on the first unlucky interval. Requiring both an absolute count and a time window (five hundred requests and five minutes, for example) removes the most common source of spurious aborts.

A minimum bake independent of signal. Some defects are time-dependent: a cache that expires after ten minutes, a scheduled job that runs hourly, a certificate or token that is refreshed on a cadence. A rollout that promotes after ninety seconds because the first window was clean has tested none of them. The bake is a lower bound on elapsed time, and it is not reducible by evidence.

A defined outcome for the inconclusive case. Many rollouts end neither promoted nor aborted: the analysis is underpowered, the traffic is too low, or the metric has drifted for unrelated reasons. The rule must say what happens then, whether that is holding at the current weight indefinitely, promoting on the strength of the previous steps, or aborting for lack of evidence. A controller with no defined inconclusive outcome will sit paused, holding two fleets and a split state, until someone intervenes.

The cost asymmetry changes the optimal threshold

Statistical convention assumes false positives and false negatives are equally undesirable. In a rollout they are not, and the direction of the asymmetry depends on the deployment technique.

For a canary carrying a small share of traffic on a service with capacity to spare, continuing too long is relatively cheap and aborting is expensive: an abort discards a bake, requires a rebuild, and consumes the appetite for automation. The threshold should be loose, and the exposure is bounded by the fraction of traffic and the time to detect. For a blue-green cutover at a hundred percent, or for a schema change already applied, continuing too long is the expensive branch and aborting is nearly free. The same service can therefore need two different rules for two different techniques, which is an argument for expressing the rule per rollout rather than per service.

The asymmetry also applies to the actions available at the boundary. An abort at a fifty percent weight means half the fleet’s requests were served by the bad version for the whole observation period, whereas an abort during a stepwise progression at one percent means almost nothing was. Step sizes are thus part of the decision rule, not a separate convenience setting: a rule with a fast reaction can afford larger steps, and a rule with slow reaction must compensate with small ones. This is why automated progressive delivery couples the analysis interval to the step schedule rather than exposing them as independent knobs.

Where the mechanism becomes the controller

Everything above describes a policy. Making the policy execute requires a control loop that owns the rollout state, applies the steps, evaluates the conditions, and terminates in one of the defined outcomes — a reconciler rather than a script, for the reasons covered in progressive delivery controllers. The metrics it evaluates must be attributable to a version, which is a property of the routing layer rather than of the analysis: without per-version labels from weighted canary routing, even a correct decision rule has nothing to compare.