Grouped Splits When Observations Are Not Independent

A dataset of medical records usually has many rows per patient. A dataset of sensor readings has many rows per device. A dataset of images has many near-duplicate frames from the same scene. In each case the row is not the sampling unit; the subject is, and a split that treats rows as independent puts the same subject on both sides of the partition. The model then memorises the subject rather than the pattern, and the score reflects a task easier than the one the model will face.

How subject overlap inflates a score

Consider a dataset of 200 patients with 50 rows each, and a target that is partly a patient-level constant, such as a baseline offset in a measurement. Under a random 5-fold split, each patient contributes about 40 rows to the training block and 10 to the test block. A model that can encode patient identity, through an identifier column, through a combination of quasi-identifying features, or through sheer capacity on a small feature set, can fit the offset for each patient from the training rows and reproduce it on the test rows. The test error is then the within-patient error, not the between-patient error, and the two can differ by an order of magnitude.

The general statement: if observations are correlated within groups, the effective sample size is closer to the number of groups than to the number of rows. A dataset with 200 groups of 50 rows has an effective sample of roughly 200 for any quantity that varies at the group level, so a test set built by row carries an uncertainty appropriate to 10,000 independent rows while the quantity it estimates has the uncertainty of 200. The interval is wrong by a factor of about $\sqrt{50}$.

This is why grouped data produces the pattern of a model that validates well and deploys badly. Nothing about the model changed between validation and deployment; the validation task was a different task, and the reported number belonged to it.

Quantifying how much the split gave away

The size of the inflation depends on how much of the target is explained by group identity. A useful diagnostic is to fit a model that receives the group identifier as its only feature and score it with the same random split. Whatever error that model achieves is the floor that any model can reach by memorising the group, and it bounds the inflation: if the identifier-only model scores far better than chance, the row split is measuring the wrong task.

A second diagnostic compares the grouped and random estimates of the same model. The gap between them is an estimate of how much the row split allowed, though it is a confounded estimate, because the two procedures also differ in the effective training size for the group-level structure. A large gap is nonetheless a signal that the random number should not be reported.

Neither diagnostic needs new data, and both are cheap relative to the fits they inform. The identifier-only model is a single fit, and the grouped estimate is a rerun of the procedure that was already being run. Skipping them means the row-split number is the only one available, and it is the one that overstates.

What a group-aware partition guarantees

A grouped split assigns every row of a group to exactly one side. The guarantee is exact: no group appears in both training and test. The cost is that the partition is now over groups rather than rows, so the achievable balance of the class distribution and of other covariates is coarser, and the test fraction can only approximate the requested value when groups have unequal sizes.

The partition problem is a bin-packing variant. Distributing groups to folds so that fold sizes are equal while class counts are also balanced is NP-hard in general, and implementations use a greedy assignment: sort groups by size descending, then place each group in the fold with the smallest current load, optionally subject to a constraint on class counts. The result is near-balanced in practice and provably not optimal. With one group holding most of the rows, no partition can balance the folds, and the honest conclusion is that the dataset supports only a handful of folds or only a single split.

Grouped k-fold then proceeds as usual: fit on all groups but one fold’s worth, score on the held-out groups, average. The estimate now answers a question about generalisation to unseen subjects, which is the deployment question whenever the unit of deployment is a subject.

import numpy as np

def group_folds(groups, k):
    """Assign whole groups to k folds, greedily balancing row counts."""
    unique, counts = np.unique(groups, return_counts=True)
    order = np.argsort(-counts)              # largest groups placed first
    folds = [[] for _ in range(k)]
    load = np.zeros(k)
    for idx in order:
        f = int(np.argmin(load))
        folds[f].append(unique[idx])
        load[f] += counts[idx]
    assignment = {g: f for f, gs in enumerate(folds) for g in gs}
    return np.array([assignment[g] for g in groups])

The greedy order is what makes the balance work: placing large groups first leaves small groups to fill the residual gaps, whereas the reverse order leaves a large group to overflow one fold at the end. Ties on argmin resolve to the lowest fold index, so the assignment is deterministic but biased toward the early folds when many groups have identical size; shuffling the unique array before the loop removes that bias at the cost of reproducibility.

Choosing the grouping key

The key is the unit that will be unseen at deployment, and that unit is not always the obvious identifier. Records grouped by patient, device, or household are the simple case. The harder cases are the ones where the grouping is implicit.

Repeated measurements of the same physical sample, split into aliquots and measured on different days, belong to one group even though they carry different identifiers. Near-duplicate records, created by a data-entry process or by augmentation, belong together because they are not independent draws. Geographic or temporal clusters behave as groups when the target varies smoothly across space or time, which is the reason spatial blocking exists: a random split of georeferenced data places neighbouring locations on both sides and lets the model interpolate.

When several plausible keys exist, the conservative choice is the coarsest one that still leaves enough groups to partition. Grouping by patient is weaker than grouping by hospital, and if the deployment target is a new hospital, then hospital is the key even though patient-level grouping is also correct. The evaluation question dictates the key: the estimate is valid for generalisation to unseen units of the grouped kind, and to nothing else.

Nested structure complicates the arithmetic. With hospitals containing patients containing visits, a split by hospital leaves patients nested within the held-out hospitals, so no patient crosses the boundary and the guarantee holds. A split by patient within hospital crosses hospital boundaries only if hospitals appear on both sides, which they do, so the estimate is for a new patient at a known hospital. Both are legitimate; they answer different questions, and a report should name which one it estimated.

The interaction with stratification and with resampling

Stratified group k-fold tries to balance class proportions across folds while keeping groups intact. The two goals conflict when a group is homogeneous in the label: a group whose rows are all positive must go entirely to one fold, so the fold’s positive count is that group’s size and cannot be adjusted. When groups are label-homogeneous, the achievable class balance across folds is limited by group sizes, and the fold-level prevalence can vary widely even though the overall prevalence is fixed.

Resampling inside a grouped workflow must respect the same boundary. Oversampling the minority class by duplicating rows before splitting places duplicates of the same row on both sides, which is leakage of the most direct kind; the duplication must happen inside each training fold, never before the split. The same applies to synthetic interpolation methods, which create new points between existing ones and therefore carry the group identity of their parents.

The general rule is that every data-dependent transformation belongs inside the fold loop. The argument is identical to the one in Stratified k-Fold Cross-Validation and the Variance of the Estimate, and grouping adds a second reason: a transformation fitted on rows from all groups learns group-level statistics, so even an unsupervised step can transmit information across the boundary.

What the grouped estimate does not tell you

A grouped estimate answers generalisation to a new group. It says nothing about generalisation to a new population of groups: if the held-out groups come from the same hospitals as the training groups, the estimate does not cover a new hospital, and no rearrangement of the same rows can produce that estimate. Covering it requires holding out at the higher level, which costs a large fraction of the data.

The estimate is also systematically pessimistic relative to a random split, sometimes by a wide margin, because it removes the subject-level information that a random split allows the model to use. That pessimism is the point: the number is an estimate of the deployment task, and the deployment task does not come with the subject’s other rows. A team comparing a grouped estimate against an earlier random-split number will see a drop and should read it as a correction rather than a regression.

Finally, grouped data makes the variance of the estimate worse. With 200 groups, a 5-fold split gives 40 groups per test fold, and the between-fold spread reflects the variation among 40 groups, which is large when group-level effects dominate. The interval arithmetic of Why a Held-Out Test Set Cannot Be Reused applies with the group count as the sample size, and the resulting intervals are often wide enough that two candidate models cannot be separated at all. That is information about the dataset, not about the comparison procedure.