Shadow Traffic: Duplicating Requests to a Candidate Without Serving Them

A candidate model that has never seen a real request has been tested against a dataset, and a dataset is a description of traffic written down at some point in the past. The distribution it encodes diverges from the traffic arriving now, and the divergence is exactly the part that a launch decision cannot afford to get wrong. The most direct way to observe a candidate on current traffic without exposing users to it is to send each arriving request to both the current model and the candidate, answer the user from the current model, and discard the candidate’s answer after recording it.

That is shadow traffic, and its appeal is that it requires no offline reconstruction of the input distribution. Its cost is that the candidate consumes real capacity, and its evidence is weaker than the name suggests, because the thing being measured is agreement with the incumbent rather than correctness.

The duplication point decides what gets compared

A request reaches the serving stack through several layers, and where the copy is taken determines what the candidate receives.

Copying at the edge proxy, before any service-specific processing, sends raw bytes: headers, body, and whatever the client sent. The candidate then has to reproduce the entire request pipeline to be comparable, including authentication-derived identity, feature lookups, and prompt assembly. This is the most faithful option and the most expensive to run, because the shadow path needs the whole stack rather than just the model server.

Copying inside the application, after features are assembled, sends the exact model input. The candidate is then compared at the model boundary, and the comparison isolates model behaviour from pipeline behaviour. Anything upstream of the copy is shared, so a bug in feature assembly affects both sides equally and the shadow cannot see it.

Copying after the model, at the response layer, compares outputs only and requires the candidate to have been driven by the same internal call. This is the cheapest and the least informative.

Most deployments copy at the second point, because the question being asked is almost always about the model. The cost is that shadow traffic cannot validate the preprocessing changes that so often accompany a model change. If the candidate ships with a new tokenizer, copying after tokenisation hides the tokenizer’s effect entirely, which is why the tokenizer belongs in the deployment identity discussed in Model Artifact Identity.

The proxy mechanism itself is a header or a route-level flag. A request carries a sampling marker, the proxy forwards it to a shadow cluster, and the response is dropped. The shadow request must be marked so that it cannot recurse: a mirrored request that the shadow service mirrors again produces exponential duplication. Marking is a header convention plus a rule at the proxy, and the rule has to be verified by inspection, because the failure mode is a capacity incident rather than a wrong answer.

Sampling, and the cases where sampling is wrong

Mirroring every request doubles model compute. For a service with headroom, a sample is enough: one request in ten, or one in a hundred, gives a comparison population large enough for most questions while keeping the extra capacity within a reserve.

Uniform sampling by request identifier is the simplest scheme and has a known bias. Requests are not uniform in cost or content. A hash on the request id spreads the sample evenly across time, which is good for drift, but rare intents are proportionally rare in the sample too. If a specific request class is 0.1% of traffic, a 1% sample yields one instance per thousand requests, and a regression confined to that class will not appear.

Stratified sampling addresses this by sampling at a higher rate within rare classes, which requires the request class to be identifiable at the proxy. Where it is not, the alternative is to mirror everything for a short window rather than a sample for a long one. A dense window of a few minutes captures the full mixture at the cost of a temporary capacity spike, and it is often the better trade because the interesting failures cluster.

Two classes of request must not be mirrored at all. The first is anything with a side effect on the user’s state: a request that mutates a record, charges a card, or advances a workflow cannot be replayed against a second backend, even one that is supposed to be read-only, unless the shadow backend is isolated from every shared datastore. The second is anything whose response depends on a nonce or a one-time token, because the duplicate consumes the token and the primary request then fails.

What the comparison can and cannot show

The shadow produces pairs of outputs. The metrics computed over those pairs are agreement statistics, and their meaning depends on the output type.

For classification, the natural metric is disagreement rate, optionally weighted by the incumbent’s confidence. A candidate that disagrees only on low-confidence cases is a different risk profile from one that disagrees uniformly. Confusion between specific label pairs is more useful than the aggregate, because it identifies the direction of the shift.

For generation, exact-match agreement is nearly zero even between two runs of the same model with sampling enabled. Comparison has to move to properties: length distribution, refusal rate, the rate at which a required structured field parses, the distribution of an embedding distance to the incumbent’s output. Each of these is a proxy, and a candidate can improve all of them while being worse.

For ranking, the observable is the ordering of the incumbent’s own output set, which understates the difference, since items the incumbent never retrieved are not in the comparison at all.

The structural weakness is that every one of these measures similarity to the incumbent. A candidate that is better in a way the incumbent does not share looks like a regression. Shadow comparison detects unintended change; it cannot detect intended improvement. That distinction determines what a shadow run is allowed to authorise: it can veto a launch, and it cannot approve one. Approval requires evidence about outcomes, which means either labelled traffic or an experiment that exposes users.

Running the candidate without letting it affect the primary

The shadow path shares hardware with the primary path unless it is deliberately separated. Three mechanisms keep the primary unaffected.

The first is a separate capacity pool with its own quota. If the shadow and the primary share a queue, a slow shadow model lengthens the primary’s queue, and the launch decision is made on degraded primary latency. Separate pools make the shadow’s cost visible and bounded.

The second is a short deadline on the shadow request. The shadow’s answer is worthless after the primary has answered, so the shadow call is cancelled at that point. Without the cancellation, a shadow model with a long tail accumulates in-flight work indefinitely. With it, the shadow’s observed latency distribution is truncated at the primary’s, which is a second reason shadow latency numbers cannot be used to predict launch latency: the shadow only ever reports the fast part of its distribution.

The third is output isolation. Shadow responses must not reach logs that feed billing, analytics, or user-visible history. Tagging the shadow request and filtering downstream is the usual approach, and the tag has to survive every hop, including asynchronous writers. Version labels on the recorded outputs matter for the same reason they do in latency measurement: an unlabelled comparison between two sets of responses cannot attribute a difference to a version, which is the problem described in Where Latency Is Measured.

def shadow_call(primary, candidate, request, budget_s):
    result = primary(request)
    # Candidate gets the same input, a bounded budget, and no path to the client.
    call = candidate.dispatch(request)      # async handle, same input
    call.cancel_after(budget_s)
    call.discard_response()                 # response is recorded, not returned
    return result

The ordering in that sketch matters. The primary call is issued first and the shadow is fired after it returns, or the two are issued concurrently with the shadow cancelled when the primary completes. Issuing the shadow first and the primary second inverts the priority and lets the candidate delay the answer.

The steady state nobody budgets for

Shadow deployments are meant to be temporary. In practice the comparison is useful enough that the shadow is kept, and the fleet then runs at roughly twice the compute for one model’s worth of served traffic. Because the shadow does not serve users, its cost has no direct owner and no latency SLO, so it is the first thing to be scaled down during a capacity crunch and the last thing to be documented. Doubling the replica count is not the only cost: the shadow pool’s replicas occupy the same nodes and draw on the same memory bandwidth, so a shadow model of the same size as the incumbent roughly halves the achievable batch concurrency on shared hardware. Separate pools with separate quotas are the only arrangement in which the primary’s capacity does not depend on whether the shadow is running.

The discipline that keeps it honest is a declared end date and an explicit comparison metric fixed before the run starts. A shadow run with no pre-declared metric accumulates whatever number looks favourable, and the decision it supports is not evidence-based. Fixing the metric in advance, along with the disagreement threshold that would block a launch, makes the run a test rather than an observation. Once the candidate is promoted, the shadow pool is deleted; keeping it running is how a temporary measurement becomes permanent infrastructure with no owner.