Confounded Comparisons Between Canary and Baseline

A canary cohort shows half the error rate of the rest of the fleet and the tempting reading is that the new build fixed something. The reading is almost certainly wrong, because the two groups differ in at least five ways that have nothing to do with the build: they received requests at different times of day, their connection pools were warmed at different moments, they may be scheduled on different hardware, they see different client versions, and every request one group handles is a request the other group does not.

The comparison is a difference, not an effect

What the telemetry provides is an observed difference between two groups. What the investigation needs is the effect of the change. The gap between them is systematic difference between the groups, and it cannot be estimated from the same data that produced the difference; it has to be controlled by design or bounded by argument.

The confounders in a canary rollout are specific enough to enumerate.

Time is the first. A rollout proceeds in batches, so the canary serves early traffic and the baseline serves late traffic, or the two alternate. Diurnal cycles in request mix are large: a checkout-heavy evening period has a different error profile than a search-heavy morning period, independent of any code. Any comparison between groups that occupy different time windows measures the day as much as the build.

Warm state is the second and is easy to underestimate. A newly started process has an empty cache, unopened connection pools, and a cold just-in-time compilation profile if the runtime uses one. Its first requests are slower and may fail differently. A canary that has been running for an hour has warm state, while a baseline instance that was restarted by the same rollout has cold state, so the direction of the apparent effect can be determined by which group was restarted last.

Infrastructure placement is the third. Unless the scheduler is known to place both groups on equivalent hosts, differences in CPU generation, memory bandwidth, disk type, or noisy neighbours are all candidates. A fleet with heterogeneous hardware produces effects between groups that are entirely attributable to the assignment mechanism.

Client and routing differences are the fourth. A canary that receives a fraction of traffic through a specific ingress rule may see a different client mix, a different proportion of bot traffic, or a different geographic distribution, and each of those changes the failure rate.

Request partitioning is the fifth and is structural. If the routing key is a user identifier, then a user who is routed to the canary stays there, and the two groups are different populations of users rather than the same population in two states. A user-level assignment is often more correct for measuring a product change and less correct for measuring an infrastructure change, where a per-request split gives more exchangeability.

Simpson’s paradox in operational metrics

The failure that arises from these confounders is not random noise. It is a systematic reversal, and the classic form is worth stating precisely because it appears in fleet metrics regularly.

A canary has a higher error rate in every individual endpoint, and a lower error rate overall. The reason is that the canary received a larger share of traffic on the endpoint with the highest intrinsic failure rate, and the aggregate is a weighted average of per-endpoint rates. Both statements are true and only one of them answers the question. The aggregate comparison is measuring the traffic mix; the per-endpoint comparison is measuring the code.

The defence is stratification: compute the comparison within groups that share the confounder’s value, then combine. Stratifying by endpoint, by request class, by host type, or by hour removes the mix effect and leaves the effect of interest. The cost is statistical power, since each stratum holds fewer observations, and the requirement that the strata be defined before looking at the result, because stratifying after inspecting the data will find a stratification that produces any desired conclusion.

How much traffic a rare-event claim needs

Sample-size intuition fails badly for low-rate events, and the arithmetic is worth doing explicitly rather than estimating. For two independent proportions $p_1$ and $p_2$, the normal approximation gives the number of observations per group as

$$n \approx \frac{(z_{\alpha/2} + z_\beta)^2 ,(p_1(1-p_1) + p_2(1-p_2))}{(p_2 - p_1)^2}$$

With a two-sided test at 95 percent confidence and 80 percent power, $z_{\alpha/2} = 1.96$ and $z_\beta = 0.84$, so the numerator’s first factor is about $7.84$. Distinguishing a one percent error rate from a one and a half percent error rate — a fifty percent relative reduction — gives $n \approx 7.84 \times 0.0203 / (0.005)^2$, which is roughly three hundred thousand requests per group. Halving the difference to a quarter of a percentage point, a twenty-five percent relative effect, pushes the requirement to about 1.2 million requests per group.

def n_per_group(p1, p2, z_alpha=1.96, z_beta=0.84):
    """Observations per group for a two-sided proportion test (normal approx.)."""
    return (z_alpha + z_beta) ** 2 * (p1 * (1 - p1) + p2 * (1 - p2)) / (p2 - p1) ** 2

print(round(n_per_group(0.010, 0.015)))   # ~305000
print(round(n_per_group(0.010, 0.0125)))  # ~1.2e6

The numbers are large and they are the reason canary analysis of rare failures is usually inconclusive. A service handling a few requests per second accumulates a hundred thousand requests in under a day; the same count on a low-traffic internal service takes weeks. Where the event is rarer still, such as a one-in-a-million corruption, the experiment is not merely underpowered but uninformative at any feasible duration, and the question has to be answered by mechanism rather than by measurement.

The requirement also grows for the difference in the opposite direction. A small improvement matters less than its confidence interval is wide over the population, and a comparison that is not sufficiently powered cannot distinguish “no effect” from “effect smaller than the study can resolve”. Reporting the interval rather than a binary verdict is the honest form of the result.

Designs that remove the confounder instead of measuring it

When the experiment can be designed rather than observed, several choices eliminate confounders structurally.

Interleaving assigns consecutive requests from the same user or session to alternating variants, so both groups see the same time window and the same client distribution. It removes the time confounder at the cost of complicating stateful behaviour, since a session that alternates between two builds sees two different caches.

Matched pairs compare hosts that are equivalent by construction, or compare the same host before and after the change. A before-and-after on the same host removes hardware and placement differences and introduces its own confounder in the form of time, which is the reason the two designs are usually combined rather than chosen between.

Stratified assignment allocates each stratum’s traffic proportionally, so that the mix is equal by construction. This requires knowing the dimensions in advance, which is the same requirement as pre-registering the strata.

The common thread is that the design decision happens before the data is seen. Once the data is in hand, no amount of analysis can create exchangeability that was absent. The one advantage a canary does have is a known onset time: the change happened when the first canary instance started serving, which bounds the time window in which the difference could have been introduced.

Interpreting a canary signal that does exist

A comparison that survives stratification and has adequate power still needs a mechanism. The difference is evidence that something differs, not evidence about what. The reduction path is the same as for any other symptom: find the population that shows the effect and the population that does not, and describe what separates them, which is a discriminating test applied to the cohort split.

The signal most easily mistaken for a real effect is regression to the mean. A canary is often chosen for investigation because its metric looked bad, and a metric that looked bad because of an unusual fluctuation will look better on the next measurement regardless of the code. Any comparison that begins with “this group looked worse, let us compare it” is subject to this, and the correction is to evaluate against a baseline measured from a population chosen before the anomaly was observed.

The signal most easily missed is heterogeneity. An effect that is positive in one request class and negative in another cancels in the aggregate, and its disappearance in the overall metric is not evidence that it is absent. Segmenting the comparison by request class, and treating a class-specific difference as a finding rather than as noise, is what turns a canary from a go/no-go gate into a source of information about the mechanism.