Health Signals for Gating a Progressive Rollout

A progressive rollout advances on evidence. The evidence arrives as a stream of measurements about the new version, and the rollout machinery either promotes it, holds it, or reverses it. Everything about how fast the rollout can move and how much damage a bad version does before it is stopped follows from which measurements are used and how quickly they change.

The most commonly used measurement is the wrong one. A readiness probe answers “should the router send traffic to this instance”, which is a question about availability, and it is answered by the instance itself. A rollout gate needs the answer to “is this version doing the right thing for users”, which is a question about outcomes and can only be answered by observing outcomes.

What each probe is allowed to mean

The probes in a container orchestrator have deliberately narrow contracts. A liveness probe that fails causes a restart. A readiness probe that fails removes the instance from the set of endpoints that receive traffic. A startup probe suppresses the other two until the process has finished initialising. Kubernetes documents these semantics explicitly, and the important consequence is that all three are instance-local: the orchestrator never consults user traffic.

Conflating liveness with readiness produces the failure that shows up as a restart storm. If a liveness probe checks a downstream dependency (a database handle, a cache connection) then a brief dependency outage fails the probe on every instance at once, and the orchestrator responds by restarting every instance, which does not repair the dependency and does remove whatever in-process caching was helping. The correct liveness signal is a property of the process alone: the event loop is running, the main thread is not deadlocked. Dependency health belongs in the readiness probe precisely because the desired response to a dependency outage is to take traffic away, not to restart.

Readiness is not a rollout gate either, but it does bound one thing: an instance that never becomes ready will not receive traffic, and a rollout that cannot get a whole replica set ready cannot progress. That is a coarse, binary filter that catches start-up failures — a missing configuration value, an unopenable socket, a failed schema check. It catches nothing about the correctness of responses.

Why the instance cannot grade itself

Any signal that the instance produces about itself shares the same blind spot: the instance does not know what the caller expected. A health endpoint that returns 200 after checking its own internals passes as long as the internals are up, even if every response body it serves is wrong. The classic shape of this failure is a version whose health endpoint queries only a shallow path (the TCP port is open, an internal flag is set) while the actual request handlers fail because a dependency client was constructed with the wrong argument order, or because a cache key format changed and every read misses a now-absent entry.

There is a defensible middle position: a readiness probe that performs a real, cheap, representative operation, such as executing a trivial query against the datastore and validating the shape of the result. This catches wiring errors that a port check misses. It still cannot catch version-specific behaviour, because the probe is written by the same people who wrote the version and tends to test what the author believed they built.

The signal that does catch behaviour is the recorded outcome of real requests: how many returned each status class, what the latency distribution looked like, whether business-level counters moved in the expected direction. RFC 9110’s separation of status classes is the useful frame here — a rising 5xx rate is an unambiguous failure, a rising 4xx rate means the new version rejects requests the old one accepted, which is sometimes intended and sometimes not, and a status distribution that is unchanged while a downstream write counter stays flat means the version is answering without doing its work. The last case is invisible to every probe.

Timing decides blast radius

The exposure of a bad version is the detection latency multiplied by the rate of affected requests. Both factors are under control, and neither is a matter of taste.

Detection latency is the sum of the measurement interval, the aggregation window, and the number of consecutive intervals required to trip the gate. A service emitting metrics every 15 seconds, evaluated over a one-minute rolling window and requiring three consecutive bad evaluations, cannot respond in under about three minutes. Whether that is acceptable is an arithmetic question: at a hundred requests per second, three minutes is roughly eighteen thousand requests served by a version that is already known to be failing by the metrics but not yet by the gate.

Aggregation windows trade the opposite way. Too short a window and the signal is dominated by noise from a handful of requests, producing aborts of healthy versions. Too long and the gate absorbs a real regression without reacting. The usual resolution is to make the window at least one order of magnitude longer than the probe or scrape interval, and to require the metric to be aggregated over a minimum absolute number of requests as well, so that a quiet period cannot trip a rate threshold on three sampled requests.

Probe frequency itself has a limit that is easy to hit: a readiness probe whose timeout exceeds its period causes overlapping evaluations, and an instance whose probe depends on a loaded dependency can then flap. The orchestrator’s failure threshold exists so that a single slow response does not evict a healthy instance, but a threshold above one multiplies the eviction latency by the number of failures required.

Choosing the baseline

A gate is a comparison, and the comparison needs a reference. Three are available, with different failure characteristics.

Comparing the canary against the stable version running at the same moment is the most statistically efficient, because both samples share the same load, the same upstream state, and the same clock. It requires per-version attribution in the metrics, which requires the router to label the requests it forwards — a header or a label attached at the split point. It fails when the two versions receive systematically different traffic, which is exactly what happens when routing is by hash on a key correlated with the workload, as discussed in weighted canary routing.

Comparing against the previous version’s own historical performance avoids the attribution problem but imports seasonality: a version released during a daily traffic peak is compared against a trough, and the reverse. Hour-aligned and day-aligned baselines reduce this, and week-aligned baselines reduce it further for services with weekly cycles.

Comparing against an absolute threshold derived from the service’s objectives is the simplest to explain and the only one that directly encodes what “bad” means. It is also the least sensitive in a healthy service: if the objective permits a 0.1% error rate and the service normally runs at 0.01%, a tenfold regression passes. Combining an absolute ceiling with a relative comparison against the stable version gets both properties.

Signals worth gating on

A small set of measurements covers most rollout decisions. The error rate by status class, segmented by version, catches outright failures. The tail of the latency distribution (the 99th percentile, not the mean, because a version that doubles the tail while leaving the average untouched is a real regression) catches performance damage; the reasons a tail is the right statistic are the same ones that make averages hide contention behind a small fraction of slow requests. Saturation signals, such as queue depth or worker occupancy, catch versions that are correct but do not keep up, and they lead the latency signal rather than following it. A small number of business-level counters (successful checkouts, completed uploads, messages processed) catch the “200 with no work done” class that no technical metric sees.

Each of these needs a threshold and a direction, and each threshold interacts with the others: a gate that aborts on any of ten independent signals at a one-in-a-hundred false-positive rate will abort a healthy rollout often enough to be turned off. That interaction, and the sample-size arithmetic behind it, is the subject of rollout analysis and decision rules. The signals are also only useful if the rollout can act on them, which requires the control loop that a progressive delivery controller provides (desired-state rollout controllers) or an operator willing to watch a dashboard, which works exactly as long as someone is watching.