Shadow Traffic and Differential Comparison Before a Cutover

A shadow deployment sends a copy of live traffic to the destination system while the source continues to serve the real response. The destination’s answers are recorded and discarded. The purpose is not to test whether the destination is fast enough, although it reveals that. The purpose is to discover, before any client is affected, the requests for which the two systems disagree.

The mechanism is easy to describe and hard to make informative. Copying the request is the simple half. Deciding whether two responses agree is the half that determines whether the exercise produces a finding or a pile of noise.

Mirroring without changing the response path

A request can be mirrored at several points, and the choice determines what the copy contains.

At the load balancer or ingress, the request is duplicated after parsing and before forwarding. The copy carries the same method, path, headers, and body. The source’s response goes to the client; the copy’s response goes to a recorder. This placement sees every request that reaches the service and requires no change to the application.

In the client library, the request is duplicated before the connection is opened. This placement can also duplicate requests that never reach the ingress, which is occasionally useful and usually a source of duplicates.

Inside the application, a handler is invoked twice. This is the least faithful option, because the handler may have already consumed state, and the second invocation may see a different cache or a different transaction.

The ingress is the right default. Its weakness is that it sees the request as the network delivered it, which for an HTTP/2 or HTTP/3 connection means a decoded request rather than the original frame sequence. That is fine for comparing application semantics and useless for comparing protocol handling.

Mirroring must not affect the source’s behaviour. Three properties are required and each is easy to lose:

The copy must not consume a shared resource the source needs. If the destination is a database the source also uses, the mirrored write changes the source’s data. This is why shadow traffic is normally paired with a destination that has its own copy of the data, and why read-only shadowing is the safe default.

The copy must not produce side effects visible to third parties. A mirrored payment authorisation charges a card. A mirrored email sends mail. The standard mitigation is to route the destination’s outbound integrations to a stub, which means the destination is being exercised with a configuration it will not carry when it takes live traffic, which is itself a source of divergence.

The copy must not delay the source. Mirroring asynchronously through a queue decouples the two, at the cost of losing the original request context if the queue backs up. Mirroring synchronously and discarding the result is simpler and couples the two latencies.

Why the two responses rarely compare byte for byte

Two systems built from the same specification will still produce different bytes. The differences fall into categories, and only some of them indicate a defect.

Timestamps, request identifiers, and trace identifiers are generated per request. Two responses to the same request will differ in these fields by construction. A comparison must either ignore them or replace them with a placeholder before diffing.

Serialisation order. Two implementations that emit a JSON object with the same members may order those members differently. Object member order is not significant in JSON, so a comparison that diffs text will report a difference where none exists semantically.

Floating-point formatting. A value that is 0.1 in one language’s default formatter may print as 0.1 in another and as 0.10000000000000001 in a third. The underlying value may be identical. Comparing parsed numbers with a tolerance is the only workable approach.

Whitespace and encoding. Insignificant whitespace, optional escaping of non-ASCII characters, and the choice between a literal and an escaped form all differ between serialisers.

Headers that describe the response rather than its content. Date, Server, Content-Length when the body is compressed differently, and any hop-by-hop headers all vary. The set of headers worth comparing is small and specific: the status code, the content type’s media type and parameters, and any application-defined header that carries semantics.

Error shape. This is where genuine differences hide. A validation failure may produce a 400 with a field-level error list in one system and a 422 with a single message in the other. Both are defensible; the clients of the source depend on one of them. A comparison that normalises away the status code will miss exactly the change that breaks callers.

Normalising without hiding what matters

The practical harness applies a transformation to both responses before comparing, and the transformation is the specification of what counts as equivalent. Writing it down is the point of the exercise, because it forces the team to state which differences are acceptable.

A normaliser in the request path is small enough to show in full. The following is JavaScript, using only built-ins:

function normalise(body, contentType) {
  if (!contentType || !contentType.includes('application/json')) return body;

  const seen = new WeakSet();
  const walk = (v) => {
    if (v === null || typeof v !== 'object') return v;
    if (seen.has(v)) return '[cycle]';
    seen.add(v);

    if (Array.isArray(v)) return v.map(walk);

    const out = {};
    // Sort members so serialiser ordering does not register as a difference.
    for (const key of Object.keys(v).sort()) {
      if (key === 'requestId' || key === 'timestamp' || key === 'traceId') continue;
      out[key] = walk(v[key]);
    }
    return out;
  };
  return JSON.stringify(walk(JSON.parse(body)));
}

Three decisions are encoded there, and each is a judgement call that should be recorded rather than assumed. Dropping identifier fields is safe because they are defined as unique per request. Sorting members is safe because the format does not order them. Dropping timestamps is safe only if no client depends on the exact value, which is worth checking against the source’s own behaviour.

What the normaliser cannot do is decide semantic equivalence for numeric fields with different precision, or for strings that differ in case. Those need per-field rules, and a harness that grows per-field rules indefinitely is a sign that the two systems genuinely differ in observable behaviour, which is a finding rather than a nuisance.

Comparison rates and what a mismatch means

The output of a shadow run is a set of requests for which the normalised responses differ. The rate is meaningless without the denominator and the sampling: a run that mirrored one request in a thousand and found two mismatches has observed a rate of two in a thousand only if the sampling was uniform and the mismatching class is not systematically under-sampled.

The mismatch classes matter more than the count. Three are worth separating:

Deterministic mismatches occur for every request of a given shape. These are specification differences and each one is a decision: change the destination, change the caller, or accept the difference and document it. A deterministic mismatch on a rarely used endpoint is a latent break for whoever uses it next.

Data-dependent mismatches occur for specific inputs. These usually indicate a difference in validation, rounding, or default handling. They are the most valuable finding, because they are the ones that a code review of the destination cannot predict.

Nondeterministic mismatches occur for the same input on repeated runs. These indicate that one of the two systems has state the other lacks: a cache, a sequence, a clock-dependent branch. They are not necessarily defects, and they make the comparison harness unreliable until the state is identified and controlled.

A run that produces only the third class has not established equivalence; it has established that the comparison method is not yet adequate. Treating a high nondeterministic rate as noise and proceeding is the specific mistake that makes shadow testing worthless, because the noise floor hides the deterministic mismatches underneath it.

What shadow traffic cannot establish

The method compares responses to requests that the source actually received. It therefore cannot exercise inputs the source never sees, which includes the inputs most likely to break a reimplementation: malformed requests that the source’s parser rejects, requests for endpoints nobody calls, and requests whose volume in the shadow window was too low to be sampled.

It also cannot establish write correctness. A mirrored write is discarded, so the destination’s resulting state is never compared against the source’s. Verifying that requires a different mechanism: applying the same writes to both and comparing the resulting stores, which is only possible when both can be reset cheaply. Where it is possible, it is far stronger evidence than response comparison, because it tests the state transition rather than the representation.

Finally, the shadow window is a sample of one period of the system’s life. A monthly report, a quarterly close, or a seasonal peak will not appear in a two-week window. The set of inputs exercised is bounded by the window’s coverage, and the plan should state which periodic behaviours were not covered rather than implying the window was representative.

The result of a shadow run feeds two downstream decisions. Deterministic mismatches become work items that must be resolved before the cutover sequence can proceed, and the set of requests never exercised becomes the list of paths where post-cutover reconciliation must look hardest. Neither is a reason to skip the exercise; both are the reason it is worth running before any client is redirected.