How Many Requests a Canary Gate Needs Before Its Verdict Means Anything

A canary that has served thirty requests without an error is not evidence of health. It is evidence that the failure rate is probably below something like one in ten, which is a statement compatible with a build that fails for one user in twenty. The gap between what a gate reports and what it establishes is a sample-size problem, and it is the single most common reason a canary passes and the fleet then breaks.

The quantity a gate is really estimating

Error rate is a proportion: out of n requests, k failed. The observed value k/n is an estimate of an unknown rate p, and its uncertainty shrinks with n at a rate of 1/sqrt(n). Doubling the sample halves the standard error. That square root is why the intuition behind watching a service for ten minutes and calling it fine is so unreliable: the difference between a sample of 100 and a sample of 10,000 is a factor of ten in precision, not a factor of a hundred.

The standard error of a proportion is sqrt(p(1-p)/n). For a service with a 1 percent baseline error rate and a canary that has served 1,000 requests, that is about 0.3 percentage points. A canary showing 1.2 percent is therefore indistinguishable from the baseline; a canary showing 3 percent is not. The same 1,000 requests on a service with a 0.01 percent baseline gives a standard error of roughly 0.03 percentage points, and a single observed error moves the estimate by 0.1 percentage points, about three standard errors. Rare events carry a different arithmetic: each individual failure is informative, and the absence of failures is nearly uninformative.

Turning that into a sample size uses the standard two-proportion formula. To detect a change from p to p + d with roughly 95 percent confidence and reasonable power, the required sample per arm is on the order of 24 * p * (1-p) / d^2, a rearrangement of the usual normal approximation with the z values for 95 percent confidence and 80 percent power folded into the constant. The consequence is worth stating three ways:

• Detecting a doubling from 1 percent to 2 percent needs roughly 2,400 requests per arm.
• Detecting a doubling from 0.1 percent to 0.2 percent needs roughly 24,000.
• Detecting a doubling from 0.01 percent to 0.02 percent needs roughly 240,000.

The pattern is that halving the rate quadruples the traffic, because both the effect size and the variance scale with p. The same reasoning governs how long each rollout wave must dwell before it can be judged. A canary plan that is adequate for a service with a 1 percent error baseline is not adequate for a service with a 0.01 percent baseline, even though the second service is the one where a regression matters most.

// RequiredSample estimates requests per arm to detect p -> p+delta
// at about 95% confidence and 80% power.
func RequiredSample(p, delta float64) int {
    if delta <= 0 || p <= 0 || p >= 1 {
        return 0
    }
    n := 24 * p * (1 - p) / (delta * delta)
    return int(math.Ceil(n))
}

Counting requests instead of minutes

Gates are frequently expressed as a soak: hold the canary for fifteen minutes, then evaluate. Fifteen minutes is a duration, not a sample. On a service handling 200 requests per second, a canary at 1 percent of traffic sees 1,800 requests in that window, which is enough for a coarse verdict on a 1 percent error rate. The same fifteen minutes on a service handling 5 requests per second at 1 percent sees 45 requests, which is enough to detect nothing except a total outage.

The correct gate condition combines both: advance when the canary has accumulated at least N requests and at least T has elapsed. N is derived from the smallest effect the gate is meant to catch; T exists to cover faults that need time rather than volume, such as a slow memory leak, a connection pool that fills only after churn, or a cache that must be evicted before the buggy path is reached. Either condition alone is a gate with a hole in it.

Traffic share sets the conversion between the two. A canary at fraction f of a service receiving r requests per second accumulates f * r * t requests in t seconds, so reaching a 10,000-request target at 1 percent of 200 requests per second takes 5,000 seconds, well over an hour, longer than most operators are willing to wait. This is why low-traffic services cannot rely on organic traffic for canary confidence and instead need a mechanism to send the canary a deliberate share of requests, which is a traffic-shaping problem rather than a deployment one. The same arithmetic appears in reverse when deciding how quickly a wave plan can advance: each gate’s sample requirement sets a lower bound on the time that wave must occupy.

Why tail latencies resist canary testing

Error counts are Bernoulli trials, so the arithmetic above applies directly. Latency percentiles are not. A p99 is the value below which 99 percent of observations fall, and estimating it from n samples means estimating the position of an extreme order statistic. To have any chance of observing the 99th percentile at all, the sample must contain at least a few hundred observations beyond it; the effective resolution of a quantile estimate is about 1/n, so a p99 from 200 samples is determined by the second-slowest request and moves substantially whenever that request changes.

Comparing two p99 values compounds the problem. The difference between a canary p99 and a baseline p99 built from small samples is dominated by which side happened to catch a slow outlier. Detecting a genuine 20 percent regression at p99 with confidence requires sample sizes in the tens of thousands per arm, two orders of magnitude more traffic than detecting a 1 percentage point error-rate change. Gates that abort on p99 ratios computed from a few hundred requests generate mostly false positives, and the standard operational response, loosening the threshold, converts them into false negatives instead.

A workable compromise is to gate on a mean or on a high-but-not-extreme quantile such as p90, where the estimate stabilises with a few thousand samples, and to treat p99 as a signal for investigation rather than a gate. Aggregating latency into a fixed histogram also introduces a resolution floor: a histogram with 1 millisecond buckets cannot distinguish a change smaller than a bucket, and a canary whose slowest requests land in the top overflow bucket reports only that they exceeded the bound.

Sequential evaluation changes the threshold

A gate evaluated once at the end of a window and a gate evaluated every minute for the same window are not the same test. Repeated looks at an accumulating sample inflate the false-positive rate: with a fixed threshold checked at many points, the probability that the noisy series crosses it at least once grows with the number of checks. An error rate that fluctuates around the baseline will eventually poke above a 2-sigma line if the line is examined often enough.

Two responses exist. The first is to widen the threshold as a function of the number of looks, which restores the intended false-positive rate at the cost of sensitivity. The second is a sequential test, which sets two boundaries: an upper one that means “abort” and a lower one that means “this is consistent with the baseline”, and continues sampling while the statistic lies between them. Wald’s sequential probability ratio test formalises this for a known baseline rate and typically reaches a decision in substantially fewer observations than a fixed-sample test of the same strength, at the cost of needing a decision boundary per hypothesis.

The practical version is a bounded sequential design: require a minimum sample, then abort early if the cumulative error count exceeds a boundary that tightens as the sample grows, and otherwise advance at the maximum sample or maximum soak, whichever comes later. The minimum sample is what stops the gate from aborting on the first unlucky request; the boundary is what stops a slow-burning regression from consuming the entire rollout window.

// AbortBoundary returns the cumulative error count that justifies aborting
// after n observations, given baseline rate p and a tolerance factor k.
// The margin term keeps small samples from aborting on one unlucky event.
func AbortBoundary(n int, p, k float64) int {
    mean := float64(n) * p
    sigma := math.Sqrt(float64(n) * p * (1 - p))
    b := mean + k*sigma
    if b < 2 { // never abort below two observed failures
        return 2
    }
    return int(math.Ceil(b))
}

The asymmetry that decides the design

The two errors a gate can make are not equally costly. Advancing a bad canary exposes the fleet to a defect; aborting a good canary costs one redeploy and a delay. Because the second is usually cheaper, gates are commonly tuned toward sensitivity (a lower threshold and a shorter window), accepting that some rollouts will be aborted without cause. The sample-size arithmetic bounds how far that tuning can go: below the resolution of the available traffic, lowering the threshold stops improving detection and only adds noise.

That bound is what makes the traffic share of the canary a first-class design parameter rather than an afterthought. A canary large enough to answer the question it is being asked, and a gate whose required sample is stated explicitly in the rollout configuration, are the two conditions under which a green canary means something. Without them the gate is a delay with a confidence interval attached, and the confidence interval is wider than the effect it was meant to catch.