Offline Evaluation Leakage and the Metrics That Hide It

An offline metric is an estimate. It is computed on a sample that is supposed to stand in for the population the model will meet, and its value predicts deployment performance only to the extent that the sample and the population match in the ways the model exploits. Most of the ways they fail to match are leakage, and leakage’s signature is a metric that is too good rather than one that is too bad. A model that looks excellent offline and degrades on release was usually evaluated in a setting that gave it information it will not have.

Duplicate rows and the same entity in both splits

The random split is the default in every library, and it assumes the rows are independent. In real data they are not. A user contributes many rows; a product appears in many sessions; a hospital visit generates a set of claims. When rows from the same entity land on both sides of the split, the model can memorize the entity rather than the pattern, and the validation score reflects how well it memorized rather than how well it generalizes.

The magnitude of the effect depends on the number of rows per entity. With one row per entity the random split is nearly correct. With a hundred rows per user, a random split puts part of nearly every user in the validation set, and a model with a per-user memorization ability, any high-capacity model with an entity identifier as a feature, scores near-perfectly on those users while having learned nothing transferable. The correct split is by group: all rows of an entity appear on one side, chosen by hashing the entity identifier so the assignment is stable across runs.

Grouping also matters for near-duplicates. Two rows describing the same event from two sources, or two versions of the same document, are different rows with the same content. The split cannot know unless the duplication is detected, which is a deduplication step in the assembly rather than a property of the splitter. Character-level or embedding-level near-duplicate detection before splitting is the only reliable defence, and its absence is the reason some benchmark scores are not comparable across papers that deduplicate and papers that do not.

Time: the split that matches deployment

Models served in production are trained on the past and evaluated on the future. A random split evaluates on a mixture of past and future, and that mixture can be much easier than the future alone. The mechanism is the same one that motivates point-in-time joins: features computed with hindsight carry traces of outcomes that the validation rows’ models would not have had.

Two temporal designs are useful. The holdout takes the last $k$ periods as validation and trains on everything before, which mirrors deployment exactly and gives one estimate with no error bars from resampling. Rolling-origin evaluation, sometimes called walk-forward validation, repeats the holdout at several cut points and averages, which gives a variance estimate and reveals whether performance is stable or dependent on the particular period chosen. The cost is $k$ times the training compute, which is why rolling evaluation is usually restricted to the final model selection rather than to every hyperparameter trial.

The subtlety with temporal splits is the gap. If labels take seven days to mature, the validation period must begin at least seven days after the training period ends, or the training rows’ labels are computed from an observation window that overlaps the validation period. Without the gap the model is trained on outcomes that partly determine what happens in validation, and the estimate is optimistic in a way that is invisible unless the label horizon is written down and checked.

Leaky features that pass every split

Splitting correctly does not protect against a feature that is computed with future information for every row. A feature equal to “number of support tickets this user will file next month” is perfectly predictive and useless, and no split design will reveal it, because the leakage is in the feature rather than in the assignment.

The available checks are structural. The first is a feature availability audit: for each feature, the earliest timestamp at which its value could have been known, compared with the prediction time. This is the discipline described in point-in-time correctness in training set assembly applied to evaluation, and it requires lineage, not statistics.

The second is a permutation test on the label. Shuffle the labels while keeping features and their timestamps intact, retrain, and evaluate. A correctly built pipeline yields a model at chance. A pipeline with leakage yields above-chance performance even on shuffled labels, because some feature still encodes the original outcome. This test costs one training run per feature group and localizes the leak to the group whose removal restores chance performance.

The third is a per-feature ablation on a small validation set: drop one feature group at a time and observe the change in the metric. A feature whose removal costs a large fraction of the metric, in a domain where no such effect has a plausible mechanism, is a candidate for a leak.

Metrics that mask the failure

The choice of metric interacts with leakage. Accuracy on an imbalanced problem is dominated by the majority class and moves very little either way, which makes a leak’s effect invisible until deployment changes the base rate. A ranking metric, precision at a fixed cut-off, or area under the ROC curve, is more sensitive to a leak because a leaked feature improves the ordering across the whole score range. Recall at a fixed false-positive rate is more sensitive still, because it is evaluated where the decisions are actually taken.

Aggregating a single number over the whole validation set also hides heterogeneity. Segmentation by a feature the model uses produces a picture of where performance comes from: a model whose overall metric is respectable but whose performance on new users is at chance has learned to recognize users rather than to generalize. The segment that corresponds to the deployment’s cold-start condition is the one that matters, and it is rarely the largest.

Confidence intervals belong in the report. A metric computed on $n$ independent samples has a standard error that for an AUC of 0.5 is roughly $1/\sqrt{n}$ in the worst case; comparing two models whose metrics differ by less than twice that is comparing noise. Reporting the interval forces the question of whether the improvement is real, and it is the difference between a metric and a measurement.

Evaluation as a regression test

Once a split design is chosen and validated, the evaluation set should be frozen and versioned in the same manner as training data. A frozen set that never changes is eventually stale, so the pairing that works is a fixed evaluation snapshot per model generation plus a fresh temporal slice for each candidate. Comparing a candidate against the incumbent on the same fresh slice is the only comparison that predicts the deployment decision.

The frozen set then serves a second purpose: detecting that a change to the training pipeline altered model behaviour. Scoring the previous model and the new one on the identical fixed snapshot isolates the change in code from the change in data. How that comparison feeds the decision to promote a model version is the subject of model registries and promotion gates.