Why a Held-Out Test Set Cannot Be Reused

A model scored on the rows it was fitted to reports a number with no interpretation outside the sample. Training minimises average loss on those rows, so the resulting average is the value the optimiser was pushed toward, not a prediction of what happens on unseen rows. The gap between the two is the reason evaluation exists as a separate discipline from fitting, and the reason a held-out set is the default instrument.

What the test-set score estimates

Write the population risk of a fixed predictor $f$ as the expected loss over the data-generating distribution, and the test-set score as the average loss over $n$ rows drawn from that distribution. If the rows are independent draws and the predictor was fixed before the rows were examined, the test average is an unbiased estimator of population risk. Unbiasedness here is a property of one estimate, not of a procedure: it says the estimator has no systematic offset, not that the number is exact.

The variance follows from the loss being an average of $n$ roughly independent terms. For a 0/1 loss with error rate $p$, the standard error is $\sqrt{p(1-p)/n}$. At $p = 0.1$ and $n = 1000$ that is about 0.0095, so a 95 percent interval spans roughly $\pm 0.019$. Two models whose test errors differ by half a percentage point on a thousand rows are not distinguishable by that measurement. This is not a defect of the split; it is the resolution of the instrument, and it caps how finely a single test set can compare candidates.

The unbiasedness claim carries a precondition that is easy to lose: the predictor must be fixed. Any decision made after reading the test score, including the decision to keep the model that scored best, invalidates the interpretation of that score as an estimate of that model’s risk.

The selection effect, and why a second look costs more than it seems

Suppose $k$ candidate models are each scored on the same test set, and the best score is reported. Each score is the true risk of its model plus noise of roughly standard deviation $\sigma$. The maximum of $k$ noisy values sits above the mean of the corresponding true risks, and the overshoot grows with $k$. For approximately Gaussian noise the expected maximum of $k$ draws exceeds the mean by about $\sigma\sqrt{2\ln k}$; at $k = 100$ that factor is near 3, so with $\sigma = 0.01$ the reported best score lands around three points of error below the truth. The approximation is crude for small $k$ and for non-Gaussian losses, but the direction is exact: the reported maximum is an upward-biased estimate of the winner’s true performance.

The same arithmetic explains why leaderboard scores degrade when a benchmark is used for long enough. The published best result is the maximum over every attempt made against the benchmark, including attempts that were never written up. As the number of attempts grows, the maximum of the noise term grows, and the benchmark stops measuring capability and starts measuring how many configurations were tried.

A second look at the test set is a weaker version of the same failure. If the score is read, a feature is dropped, and the model is re-scored on the same rows, the second score is not an estimate of the new model’s risk. The rows influenced the choice of model, so the model is now a function of the test set. The only honest description of the second number is “the loss of the model selected using these rows”, which is biased low by an amount that depends on how many alternatives were considered. The test set has become a validation set.

Leakage paths that survive a correct row split

Splitting rows before fitting is necessary and not sufficient. Information crosses the boundary through any transformation estimated on the union of both sides. Standardisation is the clearest case: computing the mean and variance over all rows and then scaling train and test with those statistics makes every test row’s representation depend on the training rows and on the other test rows. The dependency is small for a mean, larger for a variance, and larger still for transformations fitted by optimisation, such as principal components or a supervised projection.

Feature selection and target encoding leak more directly. Choosing the twenty features with the strongest univariate association to the label, measured on all rows, transmits label information from the test rows into the feature set. Target encoding computed over the full table puts each test row’s own label into the value of its categorical feature. Both procedures look like preprocessing and behave like fitting.

Duplicates and near-duplicates defeat the split at the row level. If a record appears twice, once in each side, the model can reproduce the answer for that record without generalising. Deduplication before splitting, on a key that identifies the underlying entity rather than the row, is the only defence. The same argument extends to groups: many rows per subject means a random row split places the same subject on both sides, which is the subject of Grouped Splits When Observations Are Not Independent.

Ordered data adds a temporal path. A feature computed from a window that extends past the prediction time carries the future into training, and a random split over time-ordered rows mixes past and future freely; Blocked and Rolling-Origin Validation for Time-Ordered Data covers the correction.

How large the held-out set has to be

The holdout fraction trades estimate precision against training data. A common 80/20 split of ten thousand rows leaves two thousand for scoring, which gives a standard error near 0.0067 at an error rate of 0.1, against 0.0095 for a thousand rows: a factor of about 1.4 in precision for twice the scoring rows, since precision improves with the square root of the count. The cost is that the model is fitted on eight thousand rows instead of ten thousand, and on small datasets that cost dominates: the measured score then reflects a model trained on less data than the final model will see, which biases the estimate upward in error.

Precision is not uniform across the metric. A class with thirty positive examples supports a recall estimate whose 95 percent interval spans roughly $\pm 0.11$ around the observed value, from the same binomial formula. A test set with two positive examples cannot support a recall figure at all: the observed value will be 0, 0.5, or 1. This is the practical reason rare-class metrics need either a much larger test set or a resampling scheme that scores every row out of fold, as described in Stratified k-Fold Cross-Validation and the Variance of the Estimate.

Stratification helps the variance without changing the fraction. Splitting so that each side has the same class proportions removes the random fluctuation in prevalence between train and test, which otherwise contributes to score variance on small samples. The effect is largest when a class is rare and the split is small.

Where a single split stops being the right instrument

A single split answers one question: how does this fixed predictor behave on data drawn from this distribution. It does not answer how much the answer would move under a different draw, because it produces one number and one number has no spread. It does not support tuning, because tuning consumes the split. And it cannot speak to any distribution other than the one it was drawn from; a test set collected under the same conditions as the training data is structurally incapable of detecting that conditions have changed.

Repeated holdout as the middle ground

Repeating the split with different random partitions gives a spread without the full cost of cross-validation. Each repetition produces one score from a model fitted on the same fraction, and the spread across repetitions measures the partition sensitivity that a single split hides. The repetitions are independent draws of the partition, which is the property cross-validation lacks, so the spread here is closer to an estimate of the variability of the procedure.

The catch is that each repetition refits on the same fraction, so the training-size bias is repeated rather than reduced. Repeated holdout estimates the risk of a model trained on 80 percent of the rows, and the estimate does not converge to the risk of the full-data model as the repetitions grow. It converges to the right answer to a slightly wrong question, which is usually acceptable and should be stated.

The compute is the dial. Ten repetitions of an 80/20 split cost ten fits, the same as 10-fold cross-validation, and the resulting spread has a clearer interpretation because the partitions are disjoint in the sense that matters: they were drawn independently. When the model’s fit is expensive and the dataset is large, five repetitions of a 90/10 split are often a better use of the budget than a single 5-fold loop, because each fit uses more data and the spread is measured rather than inferred.

What the alternative schemes buy

Each alternative replaces one of those missing properties. Cross-validation buys a spread estimate at the cost of $k$ fits. Nested cross-validation buys an unbiased estimate of a procedure that includes selection, at the cost of a second loop, as covered in Nested Cross-Validation and the Optimism of Tuned Scores. What none of them buy is immunity to leakage: every scheme assumes the rows are independent draws and that no transformation has seen the labels it is being scored against, and no amount of folding repairs a violation of either assumption.