Transformer State and the Fold Boundary

A preprocessing step is a function fitted to data. Once that framing is accepted, the question of where it runs becomes structural: anything fitted is a model, and a model must be fitted on the training partition of every split, not on the table that the split came from.

The state a transform carries

Each common transform has a small fitted state, and the state is what leaks.

A standard scaler carries a mean and a standard deviation per column, so 2d numbers. A median imputer carries one median per column. A quantile transformer carries a sorted sample of the column, which for a dense implementation is a fixed number of quantiles, and for an implementation that keeps the full sorted column is O(n) values. One-hot and ordinal encoders carry the category vocabulary. A target encoder carries a lookup table from level to number plus the prior. A variance filter carries the subset of retained column indices, which is the most damaging kind of state because the selection is a decision about relevance, not a summary of scale. Principal components carries the loading matrix, d * r numbers, and in some implementations the full covariance.

Two properties distinguish these. The first is the strength of the dependence on the label: mean, median, and vocabulary are label-free, while the filter indices and the target-encoder table are functions of the label. The second is the fraction of the data the state is computed from: a median over n rows is a robust statistic that moves barely when a fold is withheld, which makes its leak small in absolute terms.

Small is not zero, and the two kinds of cost compound differently. A label-free transform fitted on the full data leaks the covariate distribution of the held-out rows; the model can, in principle, learn something about the validation covariates. With a scaler this is second-order and in practice negligible. A label-dependent transform fitted on the full data leaks the held-out labels through the transform’s own mapping, and the leak is first-order: a univariate selector that keeps the twenty most correlated columns on the full table retains, by construction, the columns whose correlation includes the validation rows’ labels. The model then validates on a feature set chosen with knowledge of the answers.

Fitting the split first, then everything else

The operational rule is that the fold split is the first operation performed on the label vector, and no statistic that touches the label may be computed before it. Implemented naively, that ordering looks like careful bookkeeping and in practice it is not.

import numpy as np

def cv_score(X, y, folds, make_model, preprocess):
    scores = []
    for j in np.unique(folds):
        held = folds == j
        # Only the training rows reach preprocess; held-out rows stay untouched
        X_tr, X_va = preprocess.fit_transform(X[~held]), X[held]
        state = preprocess.state
        X_va = preprocess.transform_with(state, X_va)
        model = make_model().fit(X_tr, y[~held])
        scores.append(model.score(X_va, y[held]))
    return np.array(scores)

The detail that makes this code correct and the common alternative wrong is the line that applies the frozen state to the validation matrix. A call that fits and transforms again, or that transforms the full matrix before slicing, computes the validation columns’ statistics from the validation rows.

There is a further trap inside “fit then transform”. For a transform whose output for a row depends on the full column, such as one that ranks within the column or one that normalises over the column’s maximum, transforming the full matrix and then slicing gives each training row a value computed partly from held-out values. The row’s own column statistics are the same either way, but the rank normalisation is global and the slicing happens after. Fit on the training rows, and transform each partition with the frozen state, and the two paths coincide.

Compose the whole chain, not its parts

The unit that must cross the fold boundary is not the scaler or the encoder; it is the composition. A pipeline object holds an ordered sequence of transforms and a final estimator, and cross-validating the pipeline refits every transform on each training fold and applies them in order to the validation fold. This is the only arrangement in which the number of fit calls equals the number of folds times the number of transforms, with no transform claiming to be fitted once for all folds.

Nesting matters when a transform’s parameters are themselves chosen by cross-validation. A shrinkage strength for a target encoder, a number of components for a decomposition, or a quantile count is a hyperparameter of the transform, and if it is selected on the full data then the selection is a label-dependent decision made outside the fold. The correct arrangement selects it inside each outer training fold, which means an inner cross-validation loop; the total work is the product of the two loop lengths times the cost of a fit, and for models whose fit is expensive this is often the dominant cost of the experiment. A cheaper approximation is to fix the transform’s hyperparameters by subject-matter reasoning and accept the suboptimal value, which avoids the nested loop while keeping the fold boundary intact. The trade is explicit: the first option is closer to honest, the second is affordable.

The refit at the end

Cross-validation produces an estimate, not the model that will be used. Afterwards the pipeline is refitted on all training data with the selected configuration, and that refit recomputes every statistic on the full training set. It is not a leak to do this, because no held-out data exists at that point; the estimate was already produced. It is a common source of confusion that the final fitted scaler differs from the scaler inside every cross-validation fold, and that the final model’s in-sample error is smaller than the cross-validated estimate. The cross-validated number is the one to report, and the final refit’s in-sample number is not an estimate of anything.

The refit does introduce a subtler hazard when the encoding table or the retained columns change size between folds and between the refit. A pipeline with a variance filter selected 1,900 of 2,000 columns in folds 1 through 4 and 1,850 in fold 5 has produced five models of different dimensionality, which is legitimate; the refit selects its own subset from all rows. Comparing the fold models’ coefficients against the refit’s coefficients is therefore not a valid stability check. What is comparable is the predictions, evaluated on data none of them saw.

Diagnostics that locate the boundary

Three checks find the boundary violations without reading the pipeline.

Shuffle the labels before the split and run the entire pipeline, including all tuning. The expected score is chance. If the pipeline selects features, tunes a transform, and fits through an inner loop, the shuffled run should still come out at chance; a score above it localises the leak to whatever state was shared across folds.

Compare the cross-validated score against the score obtained by fitting once on the training set and evaluating the same model on the same validation fold. They should agree within the noise of the fold. A systematic gap of a few points means the validated model saw state from the validation rows, and the magnitude of the gap is an estimate of the leak’s contribution.

Inspect the transform’s fitted state directly for its size and its provenance. A feature selector that retains exactly the number of columns it was told to retain, chosen from a table of 30,000, has made a decision from the label; a vocabulary that grew to include every level in the validation fold has seen the validation fold. The state’s domain is the diagnostic. The same reasoning underlies the distinction between construction and partition leaks in how a feature becomes a leak.

Time and groups are outside this loop

The fold loop treats rows as exchangeable. When they are not, the partition that crosses the boundary must respect the dependence structure: contiguous time blocks, or whole groups. A pipeline that is perfectly fold-correct still leaks if the split puts one entity’s rows on both sides. That case is treated in group and duplicate leakage, and the case where the state must additionally be frozen at an event time is treated in point-in-time feature assembly.

One more property of cross-validated estimates is worth keeping in view when comparing pipelines. Repeating k-fold cross-validation with different random partitions reduces the variance of the estimate but not its bias, and the partitions must be reused across the pipelines being compared, since a difference of one partition draws can move the estimated difference more than the true difference between two configurations. Paired fold-wise differences are the right statistic for comparison, and they need the same folds on both sides.