How a Feature Becomes a Leak

A feature that contains the answer is a leak whether or not the label is among its columns. The useful definition is not about which column a value came from but about dependency: an evaluation is leaked when the value of a training example’s features depends on information that will not be available, or will not carry the same relationship, at prediction time. That dependence can hide in a derived feature, in a preprocessing statistic, or in the order of rows.

A worked construction

Take a binary classification problem on 1,000 rows with 5 percent positive rate, and build a feature f defined as the row’s index in the source table. Train any model with enough capacity to memorise that feature and validate with random k-fold cross-validation. The validation score will be well above chance on data the model has never seen, because the fold split is random and the feature is a bijection onto row identity, so each validation row’s f value coexists with its label in the training folds’ index. Replace the index with a hash of the row’s memory address or its file offset and the leak survives unchanged.

Nothing about f mentions the label. The dependency is created by the splitting procedure: any feature that identifies a row makes the fold partition irrelevant, and the partition was the entire basis for estimating generalisation. This is the general shape of every leak. The model’s validation error estimates the error on a population that includes rows it effectively trained on, and the estimate is optimistic by an amount that grows with the strength of the identity signal.

A diagnostic that separates the two cases: retrain on the same data with the suspicious feature and with it replaced by a random permutation of itself. Permutation destroys any relationship to the label while leaving the marginal distribution, the missingness pattern, and the type intact. If the validated score drops sharply under permutation, the feature carries label information; if it does not, the feature is inert and the concern is elsewhere.

The three mechanisms

Leaks are created in three places, and the remedies differ.

Construction leaks. The feature is computed from data that includes the target’s own future or outcome. A hospital readmission model that includes “number of follow-up visits in the next 30 days” is not a feature of the admission; it is a feature of the outcome. A credit model that includes a collections flag populated after default is the same error. There is no statistical fix once such a column exists; the column has to be removed or the prediction time has to be moved to after the column becomes available, which changes the problem statement.

Aggregation leaks. The feature is a statistic over a group that includes the row being predicted. Encoding a customer’s level with the mean label of all rows sharing that level puts the row’s own label into its own feature. With groups of size m the leak is proportional to 1/m within the group, so it is severe for rare categories and mild for common ones, which is why it survives casual inspection: overall metrics move a little and the leak concentrates in the tail. The remedy is an estimate that excludes the row, given in out-of-fold target encoding.

Partition leaks. The split itself fails to separate rows that are not independent. Duplicate rows, several records per patient, windows from one time series, and images of the same scene all mean that a validation row has a near-twin in the training set. The fold assignment is uniform at random over rows, so the twins are on both sides, and validation measures interpolation between near-identical points. With duplicates the effect is large: a dataset in which 20 percent of rows are duplicated has 20 percent of its validation set answerable by copying. Grouped and temporally ordered splits are the remedy.

The three are not exclusive. A duplicate pair also shares any group aggregate, which compounds the optimism.

Why training error cannot detect it

Consider any leak that raises the validation score. It also makes the training problem easier, because the same information is present in training. The model fits the training data better; the training loss falls. So the symptom is never an anomalous training error. The symptom is a validation error that is low and does not degrade as expected when the model is simplified, or a validation error that is much lower than the error on genuinely new data.

That asymmetry is the reason leakage is diagnosed by looking at the difference between reported performance and deployed performance, and why the diagnosis usually arrives late. The two standard checks that catch a large fraction of cases do not require deployment. The first is an ablation by permutation, above. The second is the random-label test: replace the labels with a random permutation and rerun the entire pipeline, including all feature construction and all hyperparameter search. A correct pipeline returns performance at chance, which for accuracy on balanced classes means 0.5 and for area under the ROC curve means 0.5. Any pipeline that scores above chance on permuted labels has a leak; the magnitude of the excess measures it. This test is cheap in the sense of compute and decisive in the sense of interpretation, and it fails on pipelines where feature selection, encoding, and model selection all run inside the fold. Running it after each structural change catches regressions introduced by new features.

Where leakage gets into the pipeline

The reason leakage is hard to remove is that the artefacts that create it are ordinary library objects. A standard scaler fitted on the full table, an imputer fitted on the full table, a variance filter applied to the full table, and a feature selector that keeps the top twenty columns by univariate association all see the validation rows. The magnitudes differ: a mean over n rows changes little when a fraction of them are withheld, so a scaler’s contribution is second-order. A univariate selector’s contribution is first-order, because selecting the columns that correlate with the label on the full table guarantees that the chosen columns correlate with the label on any subset, which is precisely the property the validation set is supposed to test.

The structural fix is a single object that owns both the transformation and the estimator and is fitted as a unit per fold, so that the transform’s fitted state cannot see the held-out rows. Pipeline APIs exist for exactly this and the mechanism is described in transformer state and fold boundaries. Manual implementations get this wrong because the natural code order is “load, clean, encode, split, fit”, and the encode step is where the state is fitted.

import numpy as np

def fit_transform_with_state(train, apply, **params):
    """Fit a transform on `train` only, then apply the frozen state."""
    state = {}
    def _fit(X):
        # Statistics are estimated here, from this matrix alone
        state.update(mean=X.mean(axis=0), index=np.flatnonzero(X.std(axis=0) > params["eps"]))
        return X[:, state["index"]]
    def _apply(X):
        return X[:, state["index"]]
    return _fit(train), apply(_apply)

The point of the signature is not the specific statistics; it is that the object returned by fitting holds state, that the state is derived from one matrix, and that transforming any other matrix uses only that state.

The honest-numbers consequence

Leakage does not merely inflate a metric; it changes the ranking of models. Suppose two models are compared on a leaked design, one with high capacity and one with low. The leak makes the problem partly solvable by memorisation, which favours capacity, so the higher-capacity model wins the comparison and loses on new data. The same reversal applies to feature sets and to hyperparameters: a search over regularisation strength will select the least-regularised configuration because the leaked design rewards capacity. Every downstream decision inherits the inversion, which is why a single leak can invalidate a month of tuning rather than one reported number.

There is an opposite failure worth naming. A pipeline can be made so strict that it discards information legitimately available at prediction time. Fitting a target encoder only on past data is correct; refusing to use any aggregate because some aggregate is a leak throws away the strongest feature available for high-cardinality categoricals. The distinction is availability at prediction time and independence across rows, not the shape of the computation. Group and duplicate structure, the case where independence itself fails, is treated in group and duplicate leakage, and the time-ordered case in point-in-time feature assembly.