Replaying Logged Traffic to Evaluate a Candidate Offline

A logged request is a record of an input and the output that was returned for it. When the output was a decision, the log also records which decision was made, and that record is a fact about the past that a new model cannot change. Replaying the request through a candidate produces a different decision, and the comparison between the two is the basis of offline evaluation.

The comparison is informative only under conditions that are easy to violate and hard to notice. The candidate must receive the same input the incumbent received, including any context the incumbent derived from elsewhere. The population of logged requests must resemble the population the candidate would serve. And if the decision affected what happened next, the logged data reflects the incumbent’s decision, not the candidate’s, so any outcome-based measurement is about a policy that was not run.

The log record and what has to be in it

A replayable record contains the model input as the model received it, not as the client sent it. That distinction is the whole difficulty. Between the client request and the model call, the serving path may have looked up user history, joined features, expanded a template, retrieved documents, and truncated a prompt to fit a limit. A log that stores only the client request requires the candidate’s harness to reproduce all of that, and reproducing it at replay time usually reads current state rather than the state that existed then.

Storing the model input verbatim solves the reproduction problem and creates a storage problem. A prompt with retrieved context can be several thousand tokens, and retaining every one at full volume is expensive. The common compromise is to log a bounded sample of full inputs plus a digest for the rest, with the digest used to detect that a replay is comparing against a record whose input was not retained.

The output side needs to record the decision, not just the rendered response. For a ranking system, that means the ordered list of items and their scores. For a classifier, the label and the score. For a generative model, the token sequence and the sampling parameters, because a replay at a different temperature is not a replay.

def replayable(record):
    return {
        "input": record["model_input"],        # exactly what the model saw
        "params": record["decode_params"],     # temperature, top_p, max_tokens
        "output": record["decision"],          # ids and scores, not rendered text
        "model": record["identity_root"],      # which version produced it
        "at": record["event_time"],
    }

The identity_root field is what makes the replay interpretable. Without it, a comparison between two sets of logged outputs is a comparison between two unknown models, and the difference cannot be attributed. This is the practical use of the deployment identity described in Model Artifact Identity.

Replay is a policy evaluation, and the policy changed the data

The subtlety appears as soon as the logged decision influenced what was observed afterwards. A recommender shows three items; the user clicks one or none. The click is a fact about the incumbent’s ranking, and a candidate that would have shown different items would have received different clicks. Scoring the candidate on the logged clicks measures how well it agrees with the incumbent’s choices, not how well it would perform.

This is the counterfactual problem, and its formal statement is that the logged data is drawn from the incumbent’s policy, so the candidate’s policy is off-support wherever the two disagree. Estimating the candidate’s performance requires either an assumption about how users would have responded to unseen items, or an estimator that reweights the logged data by the ratio of the two policies’ probabilities.

The importance-weighted estimator is the direct approach: each logged event is weighted by the probability the candidate would have taken the logged action divided by the probability the incumbent did. The estimate is unbiased when the incumbent’s probability is non-zero for every action the candidate might take, which is the support condition, and its variance grows as the ratio grows. A candidate that is nearly deterministic where the incumbent was exploratory produces weights that are enormous for a few events and zero for the rest, and the estimate is dominated by a handful of records.

The doubly robust estimator combines the weighting with a model of the outcome, which reduces variance when the outcome model is decent and remains unbiased when the weighting is correct. It is the standard tool for this problem and it still requires the support condition.

The condition fails routinely in practice. A logging policy that never explored an item class produces no data about how users respond to that class, and no estimator can recover it. Any candidate that would serve that class is evaluated on nothing, and the offline number is silent about the part of the change that matters most.

When the decision had no downstream effect

A large class of model calls has no counterfactual problem because the decision did not change the future. A fraud classifier’s score is compared against a threshold by a downstream system whose behaviour is logged separately; the score itself does not alter the transaction. A content moderation model’s output is reviewed by a human whose decision is recorded independently. An embedding model’s output feeds a search index that is rebuilt from the model, not from the interaction.

For these, replay is straightforward: run the candidate on the logged inputs, compare its outputs against the logged outputs or against labels, and compute whatever metric applies. The comparison is a model comparison on a fixed sample, and the only concerns are input fidelity and sample representativeness.

The distinction is worth making explicit in the logging design, because it determines whether the record needs exploration data. A call whose output feeds a decision that affects the future needs logged propensities; a call whose output is consumed by an independent process does not. Logging propensities for every call is wasteful, and failing to log them for the calls that need them is unrecoverable.

Sample selection and the drift between log and candidate

A log covers a period. The candidate will serve a later period. The population difference between the two is the same drift problem that a drift monitor detects, and replay inherits it.

The specific failure is a log dominated by a period that is not representative: a holiday, an incident, a marketing push. Replaying against such a log produces a metric that is accurate for that period and misleading as a prediction. Stratifying the log by time and by request class, and reporting the metric per stratum rather than pooled, makes the composition visible. A pooled number that is dominated by one stratum hides the rest.

The second selection effect is that the log contains only requests the incumbent served. Requests the incumbent rejected, for length or for rate limits, are absent. A candidate with a different limit would have served them, and the replay says nothing about those. Recording rejections with their reason, even though no model output exists for them, is what allows the replay harness to account for the boundary.

The third effect is more subtle: the log contains only requests that reached the model. Traffic that failed earlier in the pipeline never produced a record, so the replay measures the candidate on a cleaner input distribution than it will face in production. The inputs that break preprocessing are exactly the ones missing from the log.

Cost and the fidelity of the harness

Replaying a million logged prompts through a candidate model costs the same as serving a million requests, minus the network and the user. That is not cheap, and it shapes how replay is used: as a pre-launch gate on a sample, not as a continuous process.

The harness has two fidelity requirements that are often conflated. The first is that the candidate runs under the same conditions as the incumbent did: same batch size distribution, same numeric precision, same maximum sequence length. A candidate replayed with batch size one and served with batch size thirty-two produces different logits, because the reduction order in a batched matrix multiply differs. For most comparisons the difference is below the decision threshold, and for a comparison of two nearly identical models it is not.

The second is that the harness must not be the thing under test. A replay that reimplements preprocessing in the harness measures the harness’s preprocessing, not the server’s. The only reliable arrangement runs the candidate through the same serving code path that production uses, with the traffic source replaced by a file reader.

Reading the result honestly

The output of a replay is a table of per-example agreements and, where labels exist, per-example correctness for both models. The comparison of two correctness rates on the same sample is a paired test, and the pairing is the point: the same examples, so the difference has lower variance than two independent estimates.

A candidate that wins on the pooled metric can lose on every stratum that matters, which is Simpson’s paradox arriving through composition rather than through confounding. Reporting per-stratum differences alongside the pooled one is the guard against it.

The last thing a replay cannot do is establish that a change is an improvement in the sense that matters commercially. It can establish that the candidate is different, that it is better on the labelled subset, and that the difference is larger than the sampling error. Whether better on labels translates to better outcomes is a question about the labels, and it is answered by an experiment that exposes users, of the kind described in Canary Promotion as a Repeated Statistical Test. Replay narrows the set of candidates worth exposing; it does not replace the exposure.