Blocked and Rolling-Origin Validation for Time-Ordered Data

Random partitioning of a time series produces a score that no deployment can achieve. The model is trained on rows from after the rows it is scored on, so it can learn the level of the series at the evaluation instant from the future, and the resulting error understates the error of a forecast made without that information. The correction is to keep every evaluation block strictly later than its training block, which changes the shape of the folds and the meaning of the average.

What a random fold lets the model see

A stationary-looking series with a trend and a weekly cycle is the standard case. Under a random 5-fold split, a row on a Monday in March sits in the test block, and its training block contains Mondays in April, May, and June. A model with a day-of-week feature and any capacity to fit a local level interpolates the March value from its neighbours on both sides. The score reflects interpolation, not extrapolation, and the two are different tasks with different difficulty.

The failure is not confined to models with an explicit time feature. Any learner that can fit a smooth function of an index, including a decision tree that happens to split on a time-derived column, exploits the ordering. Even without a time column, leakage arrives through the target itself: a lag feature built as “value at previous row” is computed from a row that may be in the test block, so the test row’s own past includes test rows. The mechanics of that specific path are covered in Grouped Splits When Observations Are Not Independent, since a time series is the extreme case of dependent observations.

The consequence is not a small optimism. The gap between random-fold error and forward-error can be of the same order as the difference between a good model and a useless one, because interpolation across a gap of days is a different problem from extrapolating past the end of the data.

Expanding and sliding origins

The rolling-origin scheme fixes an origin time, fits on everything before it, and scores on a block after it; the origin then moves forward and the procedure repeats. Two variants differ in whether the training window grows.

An expanding window keeps the first origin fixed and extends the end forward with each step, so the final fit uses nearly all the data. This matches the deployment of a model that is retrained on all history, and it gives the later folds a training set size close to the production one. Its weakness is that it weights old regimes equally with recent ones: a series with a structural break ten years back carries that regime in every fit.

A sliding window keeps a fixed length and moves both ends, so each fit uses the most recent $w$ observations. This matches a deployment that deliberately forgets, and it adapts to regime change at the cost of a smaller effective sample. Choosing $w$ is a bias-variance decision in itself: short windows track change and estimate noisily, long windows estimate well and lag.

Between the two lies a hybrid in which the window grows until it reaches a cap and then slides, which is what many production retraining schedules implement without naming it.

def rolling_origins(n, initial, horizon, step, expanding=True):
    """Yield (train_end, test_start, test_end) index triples, all half-open."""
    train_end = initial
    while train_end + horizon <= n:
        yield 0 if expanding else train_end - initial, train_end, train_end + horizon
        train_end += step

The parameters interact. With $n$ observations, an initial training size $i$, horizon $h$, and step $s$, the number of folds is $\lfloor (n - i - h)/s \rfloor + 1$. Halving the step doubles the fold count and roughly doubles the compute, without adding independent information: overlapping test blocks share rows, so the folds are correlated in proportion to their overlap. A step equal to the horizon gives disjoint test blocks and the most independent folds per unit of compute; a step of one gives many folds and a smooth error curve at the cost of heavy correlation.

The gap, and why horizon length is a modelling choice

If a feature for a row at time $t$ uses the value at $t-1$, then a test block beginning at $t_0$ needs the value at $t_0-1$, which lives in the training block. That is legitimate at prediction time, because the value at $t_0-1$ is known when the forecast for $t_0$ is made. What is not legitimate is a test row at $t_0$ whose feature was computed from $t_0 + 1$, which happens when features are built over a centred window or when the training block’s last row has a lag computed from the first row of the test block.

A gap of $g$ rows between the end of training and the start of testing removes the paths that reach forward by up to $g$ steps. The gap must be at least as long as the longest look-ahead in any feature, and at least as long as the label horizon when the label for a row at $t$ is defined by values up to $t+h$: without the gap, the label of the last training row is computed from rows inside the test block, so the training set contains information about the test period’s outcomes.

The horizon is also a design parameter of the evaluation itself. Scoring one step ahead and scoring thirty steps ahead measure different capabilities, and a model tuned for the former can be poor at the latter. A useful evaluation reports the error curve as a function of horizon, since the shape of that curve, not a single number, tells whether the model’s advantage comes from persistence or from learned structure.

The baselines a forecast must beat

A time-ordered evaluation that reports only the model’s error leaves the reader unable to judge whether the error is good. Two baselines are cheap and almost always relevant. The naive forecast repeats the last observed value, and its error is the scale of the series’ one-step movement; the seasonal naive forecast repeats the value from one season earlier, and its error is the scale of the movement beyond the seasonal pattern. Both are fitted from the training block alone, so they cost nothing extra and respect the same ordering.

A model that does not beat the seasonal naive forecast at the horizon of interest is not adding value, regardless of how it compares to a random baseline. This comparison is the one the scaled error metric encodes, and it is the reason that metric exists: a scaled error below one means the model beat naive on the training block’s scale, and the number is comparable across series whose levels differ by orders of magnitude.

The baselines also diagnose feature failures. If a model with lag features matches the naive forecast exactly, its learned component is contributing nothing and the fit has collapsed to persistence; if it is worse than naive, the fit is adding noise to the baseline. Both outcomes are visible only against the baseline and invisible in a raw error number.

Metrics whose scale depends on the series

Error metrics on a series are not comparable across series with different scales, which matters when folds come from different periods. Mean absolute error on a series that tripled in level over the evaluation window penalises the early folds less than the late ones purely through scale. Mean absolute percentage error normalises by the actual value but is undefined at zero and explodes near it, which makes it unusable on series that cross or approach zero.

The mean absolute scaled error divides the absolute error by the mean absolute error of the naive one-step forecast on the training data, which makes the result scale-free and comparable across series. A value below one means the model beats the naive forecast; a value above one means it does not. That comparison is the one worth reporting, because it is the only baseline that is always available and always relevant.

Aggregation across folds has its own subtlety. Averaging per-fold metrics weights each fold equally regardless of its size; pooling all errors across folds weights by row. When the folds have different horizons or different numbers of rows, the two aggregates answer different questions, and the pooled version is dominated by whichever fold has the most rows.

Where the scheme still fails

Rolling origin assumes the future resembles the recent past. A series with a regime change inside the evaluation window will score the model on a distribution it never saw, and the resulting error says more about the change than about the model. That is a correct signal and an unwelcome one: the honest reading is that no validation scheme can estimate performance under a distribution shift that has not yet happened, and the error is a measurement of the shift’s cost rather than of model quality.

The scheme also assumes the origins are independent enough to average. Overlapping test blocks make the per-fold errors correlated, so the spread across folds is not the standard error of the mean; the same caveat as in Stratified k-Fold Cross-Validation and the Variance of the Estimate applies with more force, since the correlation here comes from shared rows rather than shared training data.

Finally, the scheme measures one model configuration per fold. Selecting hyperparameters requires an inner loop of origins inside each training block, which is the time-series form of the nested procedure described in Nested Cross-Validation and the Optimism of Tuned Scores. Skipping the inner loop and tuning on the outer folds produces the same upward bias as tuning on a test set, and the bias is larger here because the outer folds are fewer and more correlated.