Rollout Waves and the Arithmetic of Blast Radius

A deployment that replaces every running instance at once has exactly one decision point: whether to start. Everything the operator learns about the new build arrives after the old build is already gone. Staged rollout replaces that single decision with a sequence of smaller ones, each of which can be answered with evidence and each of which leaves most of the fleet untouched. The engineering content of the technique is not the sequence itself but the two quantities that define it: how much of the fleet a stage covers, and what evidence is sufficient to advance.

A wave is a set of instances plus a gate

The unit of a staged rollout is a wave: a subset of the target population that receives the new version together with a predicate that must hold before the next subset is touched. A plan is a list of such subsets, usually growing geometrically: one instance, then a few percent, then a quarter, then the remainder, with the last wave being simply “everything not yet updated”.

Wave size and wave count are related by arithmetic that is easy to get wrong. A fleet of 200 instances with a first wave of 1 percent contains two instances. That is the floor: a canary cannot be smaller than one instance, and on a small fleet the smallest meaningful wave is a larger fraction than the plan intends. The same 1 percent on a 20-instance fleet is zero instances, and most rollout controllers silently round up to one, which means the canary is 5 percent of the fleet and the blast radius is five times the stated figure.

The resolution of the rollout is set by the same number. If a wave covers k instances out of n, then the smallest failure rate the gate can distinguish from zero is roughly 1/k per instance. Two canary instances can only reveal failures that affect at least half of them; a defect that strikes one instance in twenty will pass a canary of two with high probability and appear only when the wave covers a large share of the fleet. This is the reason wave plans grow rather than shrink: early waves are cheap because they are small, and their gates are weak for exactly the same reason. The statistical side of that trade is covered in why a canary needs traffic before its health signal means anything.

Blast radius is often stated as a percentage of instances, which is the wrong denominator. What matters is the fraction of user requests that can be served by the new code, and that fraction depends on traffic distribution rather than on instance count. A wave of 5 percent of instances behind a load balancer that assigns work uniformly touches about 5 percent of requests. The same 5 percent of instances in a system where one instance holds a hot shard touches nearly all traffic for the users on that shard. Instance-weighted blast radius is a proxy that holds only under uniform load.

type Wave struct {
    Fraction float64       // share of the fleet, 0 < Fraction <= 1
    Soak     time.Duration // minimum dwell before the gate is evaluated
}

// PlanWaves converts fractions into instance counts, never dropping below one.
func PlanWaves(n int, plan []Wave) []int {
    counts := make([]int, len(plan))
    for i, w := range plan {
        c := int(math.Round(w.Fraction * float64(n)))
        if c < 1 {
            c = 1 // a canary of zero instances gates on nothing
        }
        counts[i] = c
    }
    return counts
}

The gate is a predicate over a window, not a reading

A gate compares a signal against a threshold over an interval. Three details decide whether it behaves sensibly.

The first is the baseline. An absolute threshold on error rate is meaningless without knowing what the service normally does: a service with a 0.4 percent steady-state error rate will trip a 0.1 percent gate on every deploy, and a service with a 0.001 percent rate will never trip a 1 percent gate even when the new build is an order of magnitude worse. Gates that compare the canary against the unchanged remainder of the fleet are insensitive to the service’s own baseline, but they inherit a second problem: when the canary is small, its rate is noisy, and a comparison against a large stable population will produce false aborts from sampling variation alone.

The second is the minimum sample. A gate evaluated on forty requests cannot distinguish 1 percent from 10 percent. Rollout controllers that evaluate a gate on a fixed wall-clock soak rather than a fixed request count are effectively gating on whatever traffic happened to arrive, which on a low-traffic service may be a handful of requests and on a busy one may be millions. The honest formulation asks for both: advance when the canary has served at least N requests and the interval has elapsed, and hold until both conditions are met.

The third is the treatment of missing data. A canary that receives no traffic reports no errors. A gate written as “abort if error rate exceeds 5 percent” therefore passes a canary that is not serving anything at all, including one that is crash-looping behind a load balancer that has already removed it from rotation. Every gate needs an explicit liveness condition, and the absence of data must be a hold or an abort, never a pass.

Duration matters as much as magnitude. A single evaluation window catches a fault that is present continuously; a fault that appears only under a particular request shape, or only after a cache has been evicted, may need a window long enough to cover a full cycle of the workload. Hysteresis, which requires the signal to exceed the threshold for a sustained interval rather than instantaneously, trades detection latency for a lower false-abort rate, and the two are the only knobs in that trade.

Every wave costs time, and time is the scarce resource

A plan of five waves with a ten-minute soak each takes at least fifty minutes before the last instance is replaced, plus the time to pull images, restart processes, and warm caches at each stage. On a fleet where a single instance takes several minutes to become ready, the wall-clock cost of staging is dominated by readiness rather than by the soak.

That cost is not merely inconvenient; it shapes behaviour. A rollout that takes hours encourages batching unrelated changes into one deploy, which increases the size of the change and therefore the difficulty of attributing a failure to a cause. A rollout that takes minutes encourages small deploys, which is the property that makes staged rollout worth its overhead. The design constraint is real: the total time to full deployment is bounded below by the sum of the wave soaks, and the number of waves is bounded above by the tolerance for that delay.

There is a second cost that is easy to overlook: during a mixed-version period the fleet must run two versions simultaneously, and any state they share (a schema, a message format, a cache key layout) must be readable by both. Staging a rollout therefore requires the compatibility work described in mixed-version operation during a rolling upgrade, and the longer the rollout takes, the longer that compatibility window stays open.

Correlated faults do not respect wave boundaries

The premise of staging is that a failure in the canary predicts a failure in the rest. That premise holds when the fault is a property of the new binary: a null dereference on a rare input path, a regression in a hot loop, a missing index. It fails when the fault is a property of the environment that every instance shares.

A configuration change deployed alongside the code is one example. If the new build reads a flag whose default changed, the canary and the fleet fail together, and the canary’s health is evidence of nothing. A shared dependency is another: a new build that issues ten times as many queries to a database will take the database down at 10 percent of the fleet, and the canary may look healthy while the load is small. A protocol change that the peer service does not understand fails only once both sides are new, which may not be until the final wave. In each case the wave plan is intact and the model behind it is void.

The practical consequence is that a wave plan reduces exposure to failures localized in the changed code and does almost nothing for failures that are correlated across the fleet. Systems that take rollout safety seriously separate the two: the canary is used to detect binary-local faults, and changes to shared configuration, schemas, and protocols get their own staged path with their own gates, deployed before or after the code rather than with it.

Rollback latency is part of the exposure

A wave that fails its gate does not stop affecting users at the moment the gate trips. The affected instances continue serving the new version until they are replaced, and the replacement takes as long as the rollout took to reach them. Effective exposure is therefore the failure rate multiplied by the interval between the fault appearing and the last bad instance being replaced. That interval includes gate evaluation, the decision to abort, and a full redeploy of every instance already updated.

That arithmetic changes which gate settings are defensible. A gate that requires a thirty-minute sustained breach to abort is not conservative; on a fleet where rollback takes twenty minutes, it permits fifty minutes of degraded service at whatever fraction of the fleet has been updated. A fast, noisy gate that aborts on a five-minute breach and is occasionally wrong costs one unnecessary rollback and a re-run. The asymmetry between those two outcomes is the whole argument for gating on shorter windows with more sensitive thresholds, provided the sample size supports it.

Rollback is also not always the inverse of rollout. Replacing a binary is cheap; undoing a schema migration, a backfill, or a message that other services have already consumed is not. The distinction between a change that can be reversed by redeploying and one that cannot is the subject of rolling back a change that already mutated state, and it determines whether a wave gate is a safety mechanism or merely a delay before an irreversible step.

Where the model stops

Staged rollout is a mechanism for converting a single large bet into a series of small ones. It works when failures are detectable from the signals a gate can see, when the population is large enough for the small waves to be informative, and when the fault is a property of the new code rather than of the environment. When any of those conditions fails, the plan still executes and still reports success; it simply stops providing the guarantee it appears to provide. The useful discipline is to write down, for each wave, what evidence would justify advancing and what fraction of traffic is actually exposed, and to notice when the answer to the second question is not the number in the plan.