Group and Duplicate Leakage in Cross-Validation Splits

The derivation of k-fold cross-validation assumes that the n rows are independent draws from a common distribution, so that holding out a subset leaves the rest a representative sample of the same population. Most datasets violate this, and the violation is not a small perturbation of the estimate; it can change the estimate from useless to accurate, or the reverse, without any visible sign.

What independence buys and what its absence destroys

Let the rows be partitioned into g groups of size m, with rows inside a group exchangeable and groups independent. The effective sample size is g, not n = g * m. A row-level split places each group’s rows across folds in proportion to the fold sizes, so a validation row finds, among its training neighbours, roughly (k - 1)/k of its own group, that is, about m * (k - 1)/k near-identical rows. If a group’s rows share a label, which happens when the group is the unit of the outcome, such as a patient, a customer, a document, or a scene, the validation label is recoverable from the training rows. The estimator then measures the ability to interpolate within a group, which is not what the model will be asked to do.

Two quantities make the size of the effect concrete. With k = 5 folds, a validation row has a training set containing 80 percent of its own group, and with group size m the within-group copy count is 0.8 * m. A dataset whose records are deduplicated at the entity level but not at the row level therefore has an inflation that grows with the number of records per entity. At m = 1 there is no group structure and the split is correct; at m = 10 the estimate is closer to a within-group fit than to a generalisation estimate.

There is a second, quieter effect: the variance of the cross-validated estimate. Fold scores are averages over validation rows, and rows within a group are correlated, so a fold that happens to contain several large groups has a score dominated by those groups. The variance of the k fold scores is larger than the independence assumption predicts by a factor related to the intraclass correlation and the group size, so the standard error computed as the sample standard deviation of fold scores divided by the square root of k understates the true uncertainty. The estimate is not only optimistic; it is also reported with a confidence that is not justified.

Duplicates are the degenerate case

Exact duplicates are groups of identical feature vectors. A split that puts one copy in train and one in validation gives the model the answer directly, and any high-capacity model retrieves it. The effect is easy to quantify: if a fraction d of rows has an exact twin elsewhere, then about d * (k - 1) of the validation set is answerable from training by matching, so accuracy on a balanced binary problem is at least 0.5 + 0.5 * d * (k - 1) minus whatever the model fails to match. At d = 0.2 and k = 5 the floor is 0.9 for a perfect memoriser.

Near-duplicates are the more common and more damaging case, because they are not detected by an equality check. Two images of the same scene taken a second apart, two versions of a document that differ in a title, two transactions from one automated process: their feature vectors are close but not equal, so any similarity-based model generalises between them, and the deduplication that would catch exact copies does not fire. The detection that works is pairwise similarity under the model’s own metric, or a hash of the preprocessing output, and both are more work than a group column because they require constructing the group variable.

Deduplication before splitting is the first fix and it has a cost: removing duplicates changes the target distribution, since duplicates are more frequent for common patterns, so a model trained on deduplicated data is calibrated for a different population. The usual resolution is to split by the unique entity and to keep the duplicate rows within their entity’s partition, so the model sees the duplication it will see at prediction time, while no entity straddles the boundary. The consequence is that every subsequent statistic, including prevalence, is measured over duplicated rows.

import numpy as np

def group_folds(groups, k, seed=0):
    """Assign whole groups to folds; each group appears in exactly one fold."""
    rng = np.random.default_rng(seed)
    uniq, counts = np.unique(groups, return_counts=True)
    order = rng.permutation(len(uniq))
    # Balance by row count, largest groups first, into the emptiest fold
    load = np.zeros(k)
    fold_of_group = {}
    for idx in order[np.argsort(-counts[order], kind="stable")]:
        f = int(np.argmin(load))
        fold_of_group[uniq[idx]] = f
        load[f] += counts[idx]
    return np.array([fold_of_group[g] for g in groups]), load

The greedy assignment balances the number of rows, not the number of groups. That choice matters: balancing groups evenly produces folds whose prevalence can differ substantially when group sizes vary, while balancing rows keeps the folds’ sizes comparable but can under-represent a large group in the smallest fold. Neither is universally right; what is not negotiable is that no group is split.

Blocked folds for ordered data

When rows are ordered in time, the correct partition is a set of contiguous blocks, because a random assignment of blocks leaks across the boundary in both directions: a training block after a validation block contains information that a model predicting at the validation block’s time would not have. In practice this means training on a prefix and validating on the following segment, and advancing the window.

The choice between expanding and sliding windows is a statement about non-stationarity. An expanding window keeps all history, which is right when the relationship is stable and more data always helps. A sliding window discards old data, which is right when the relationship drifts; the window length becomes a hyperparameter that must itself be selected without looking at future data, which means selecting it inside the earliest usable window and applying the choice forward. Fallback between the two, where the window starts as expanding and then slides, is common and requires care to avoid a discontinuity in the size of the training set across folds.

A practical hazard specific to blocked folds is the gap. When the features of a row depend on a trailing window of past rows, the first rows of a validation block depend on the last rows of the training block, so the boundary itself leaks: the training data’s final row contributes to the validation block’s first feature. Excluding a gap of the window length between the blocks removes the overlap, at the cost of a shorter training or validation set. The overlap is exactly the window length in rows, which is why the gap parameter is usually expressed in the same units as the aggregation window; the mechanics of building the aggregates are treated in point-in-time feature assembly.

The cost of grouping

Grouped splitting reduces the number of independent units available for validation. With g groups and k folds, each fold’s validation set contains about g/k groups, and if that number is small the fold score is dominated by a handful of entities. A dataset with 40 groups and 5 folds has 8 groups per validation fold; the standard deviation of the fold scores is then driven by group-level variation, and the confidence interval is wide enough that differences between models become undetectable. The remedy is more folds, with k bounded by the number of groups, or leave-one-group-out cross-validation, which fits as many models as there are groups and is affordable only when the model is cheap.

Leave-one-group-out has a property worth noting: each fit sees g - 1 groups, so the training set size is nearly constant across folds and the estimate is nearly unbiased for a model trained on g - 1 groups, but a model trained on all g groups has slightly less variance than any of the fold models. The direction of the bias is optimistic, and its size grows as g shrinks. With grouped data of small g, no resampling scheme produces an estimate with a small variance; the honest report is the spread, not the mean alone.

Hierarchical structure complicates the picture further. If rows nest inside groups that nest inside sites, the partition must be made at the coarsest level that the deployment will not share, which is usually the site, so that the estimate reflects performance on a site not seen in training. Splitting at the group level and validating within sites measures a different question and will be more optimistic. Deciding which level is right is not a statistical question; it is a question about what the model will be asked to do.

Verifying that the split did what it claims

Grouped splits are easy to believe they are correct and easy to get wrong when the group key is derived. Three checks are worth the effort.

Count the overlap: intersect the group keys present in each training fold with the keys in its validation fold. The intersection must be empty, and the check must be run on the key that the deployment will use, not on a proxy. A key derived from a customer identifier has an empty intersection by construction; a key derived from a postcode does not, because postcodes are shared.

Check the fold-level score spread against the row-level score spread. If grouped folds produce a much higher variance than row-level folds on the same data, the group structure is real and was being masked. A gap of several points between the group mean and the row mean is the standard signature.

Test on genuinely new entities. Where a held-out period or a held-out site is available, evaluate on it, and treat the result as the reference that the cross-validation estimate is calibrated against. The relationship between the two is the calibration error of the entire validation scheme, and it is the only check that does not rely on the assumption under test.

The mechanics that make a grouped or blocked split correct at the level of preprocessing state are the same as those for any other split, described in transformer state and fold boundaries, and the general taxonomy of how the leak is introduced is in how a feature becomes a leak. What distinguishes this case is that no amount of pipeline discipline fixes it: the split itself must change, and every statistic, from the mean imputation value upward, must be computed after the group-aware partition exists.