Weighted Canary Routing at the Edge and What the Weight Measures

A canary deployment sends a small share of traffic to a new version while the previous version serves the rest, so that a defect large enough to matter shows up before it affects everyone. The interesting part of the technique is not the percentage but the denominator: a weight of five is meaningless until it is known what is being counted, what is making the choice, and how the choice is remembered.

The answers differ enough between routing layers that the same nominal weight produces very different exposure. Getting the denominator right is what makes the difference between a canary that catches a bug in the first minute and a canary that reports nothing for a day while the bug affects everyone.

The unit a weight applies to

At layer 4 the forwarding decision is made at connect time. A TCP or UDP flow is assigned to a backend when it is established, and every byte on that flow goes to the same backend. The weight therefore divides new connections, not requests. A client that opens one connection and sends ten thousand requests contributes one slice of one connection, and its entire request volume lands on whichever version won the assignment. If the workload is dominated by a few long-lived clients (a handful of internal services with connection pools, or mobile clients holding HTTP/2 connections open) then the observed split can be nothing like the configured one at the request level, and can stay that way for as long as the connections live.

At layer 7 the decision is made per request, or per stream in the case of HTTP/2 and HTTP/3, so the weight divides requests. That is the more intuitive denominator and the one most canary tooling assumes. Even here the request-level share is only an expectation: with a random draw per request, a 5% weight on a service receiving a hundred requests during the canary period yields a binomial sample whose realised share can plausibly be anywhere from a few percent to ten percent, and the endpoints that matter most (the rare ones) will be represented by very few samples. Splitting at the endpoint or route level rather than uniformly across all traffic is often the more informative configuration, because it concentrates the sample where the change actually is.

Some routers implement weighted upstreams as relative weights among a set of backends rather than as an absolute percentage. nginx’s server ... weight= in an upstream group has exactly this semantics: weights are relative to the others in the group, and a backend that is marked down has its share redistributed among the survivors. This is where a canary’s exposure can silently change. If the canary is expressed as a subset of endpoints with a lower weight, and one of those endpoints fails its health check, the remaining canary endpoints inherit its weight only if the router treats them individually; if the canary is expressed as a second service whose endpoints are behind a single weighted reference, the weight applies to the group and survives membership changes.

Hashing turns percentages into fixed partitions

Random per-request selection gives smooth, expected splits but different users see different versions on consecutive requests. Most canaries instead hash a stable key (a session cookie, a user id header, a client address) and map the hash range onto versions. This gives the useful property that a user stays on one version for the duration of the canary, which matters when the versions differ in cached client state, and it makes the split reproducible and re-computable across independent proxies.

The cost is that the split is no longer a random sample of requests; it is a deterministic partition of keys. If the keys are not uniformly distributed (a header that is empty on a large class of traffic, an address space where one /24 dominates) the partition can be badly skewed while still being exactly consistent everywhere. Consistent hashing, introduced for distributed caching, makes the remapping cheap when a version is added or removed, but it does not improve uniformity of a skewed key.

The choice also affects what the canary can see. Hashing on a user identifier means a bug that only manifests for a specific tenant appears only if that tenant’s key falls in the canary bucket. Targeting the canary by an explicit header or by internal accounts avoids the lottery, at the cost of exercising a narrower slice of real behaviour.

What a small share can and cannot resolve

It is tempting to reason that if the new version has a problem affecting one request in a thousand, then one percent of traffic will surface it. The arithmetic is less kind. With a 1% split, the expected number of affected requests after a thousand total requests is 0.01; several tens of thousands of total requests are needed before the affected count reaches single digits. Meanwhile the same period has produced tens of thousands of unaffected canary requests, which is more than enough to establish that the common paths work.

That asymmetry is the useful mental model. A canary is a filter for failures with a high per-request probability, such as a crash on start-up, a missing dependency, a serialisation mismatch on a frequently used route, or a large latency regression. It is a poor filter for rare-path defects: an error in a code path taken by one request in a million, a bug specific to a rarely used locale, or a subtle wrong-value write that only some inputs trigger. Those need explicit targeting (route a named account or a synthetic workload at the canary), a longer bake, or a different instrument — the statistical decision rules discussed in rollout analysis and decision rules.

Rate-limited services make this worse. If the service receives a few hundred requests an hour, a 5% canary receives a handful, and the observation window needed to distinguish a working version from a broken one is measured in days. For these, percentage routing is not the right tool: either the canary is exercised by synthetic traffic that names a canary-specific route, or the version is rolled out with an immediate error-budget gate, since the fleet is small enough that the blast radius is bounded anyway.

Where the weight is stored and how it changes

Weights live in whichever object holds the routing rules: an nginx upstream group, a Gateway API HTTPRoute backend reference with an explicit weight, a mesh virtual service with two destinations and weights summing to a hundred, a managed load balancer’s target group weights. Changing one is a control-plane write; the dataplane applies it asymmetrically, because each proxy in the path learns of the change when it next receives configuration. During that interval different proxies apply different weights, so the instantaneous global split is not reproducible even when the configured value is.

Two consequences follow. First, a change in weight is not a step function at the request level, and observations taken during the transition mix two configurations. Second, the analysis of canary results must attribute requests to the version that served them, not to the weight that was configured at the edge — a per-version label or a header added by the proxy is what makes the comparison possible at all. Where the canary is chosen by hash rather than by random draw, label-based attribution is straightforward, since the proxy knows which bucket it selected.

Failures that the split itself causes

Some canary problems come from the division of traffic rather than from the new code. Caches and connection pools sized for a single fleet are now split across two, so both halves run colder. Retries at the client or the proxy can send a request to the stable version after it fails on the canary, which hides the failure from user-facing metrics while it remains visible in the canary’s own error count — and can duplicate side effects if the request had already been applied. Background and scheduled work is usually not part of the split at all, so a canary can pass every check while the new version’s job processor is untested; conversely, if both versions process the same queue, the canary processes a share determined by consumer count, not by any weight.

Finally, the traffic that reaches a canary is rarely representative. Health checks, synthetic probes, and internal service-to-service calls may or may not be included depending on where the split sits in the request path. Internal calls between services in the same mesh are usually split too, which multiplies the canary’s effective exposure across a request graph — a user request touching five services has five chances to hit a canary. That interaction is one reason canaries are usually paired with a layer that decouples code paths from traffic, as described in feature flags versus traffic routing, and with health signals that measure the outcome rather than the deployment, covered in health signals for rollout gating.