Point-in-Time Correctness in Training Set Assembly

Assembling a training table looks like a join and is not one. The left side is a set of labelled events, a user clicked, a transaction settled, a parcel arrived, and each row needs the feature values that were knowable at the moment the label’s outcome began. An ordinary equality join on entity key returns one value per entity per feature, typically the latest, and the latest is by definition computed from events that may lie on the far side of the outcome. The model then trains on a feature that contains a trace of what it is supposed to predict.

The discipline that prevents this is called a point-in-time join or an as-of join: for each labelled row, retrieve the feature value whose validity interval contains the row’s anchor timestamp, using only data that was available then.

Why ingestion time and event time both appear

Every event carries at least two timestamps. The event time is when the thing happened; the ingestion time, or processing time, is when the pipeline learned about it. They differ by the latency of the source, which for a mobile client that queues writes offline can be hours or days, and for a batch feed can be a day.

The training join must use event time, because the model will be served against the world’s event order, not the pipeline’s. But the snapshot of the data available for a historical reconstruction is bounded by ingestion time: when rebuilding the feature value for a label at time $t$, a correct reconstruction may only use events with ingestion time before $t$ as well as event time before $t$. An event that happened at $t - 1$ hour but arrived at $t + 6$ hours was not visible at inference time, so including it leaks. Late data is the mechanism by which event-time-correct joins still produce unrealistic training rows.

The Dataflow model names this exactly: watermarks estimate how far event time has advanced, and allowed lateness determines how long a window stays open for late arrivals. A training join that ignores the ingestion cut is equivalent to a pipeline with unbounded allowed lateness, which produces a value no live system could have produced at that moment.

The join itself

This fragment of SQL sketches the shape, using a features table keyed by entity with validity intervals rather than a single row per entity:

-- Each labelled row is anchored at label.as_of; take the feature row whose
-- validity interval contains that instant.
SELECT l.entity_id, l.as_of, l.label, f.value
FROM labels AS l
LEFT JOIN LATERAL (
    SELECT value
    FROM features
    WHERE entity_id = l.entity_id
      AND valid_from <= l.as_of
      AND (valid_to IS NULL OR l.as_of < valid_to)
    ORDER BY valid_from DESC
    LIMIT 1
) AS f ON TRUE;

Three details decide whether this is correct in practice.

The interval convention must be half-open, [valid_from, valid_to), or two adjacent rows both claim the boundary instant and the LIMIT 1 picks arbitrarily. Half-open intervals partition time without gaps or overlaps, and they make the ORDER BY deterministic.

The LEFT JOIN is not optional. An inner join silently drops entities with no feature value, which is a selection bias: the model trains on the subset of entities that already had data. Whether a missing feature should become zero, null, or a separate indicator column is a modelling decision, but it has to be made in the pipeline rather than by the join’s default.

The indexes must support the interval predicate on (entity, valid_from), or the lateral subquery is a scan per row. The cost is roughly a binary search plus a fetch per labelled row, so the assembly time scales with the number of labels rather than the product of labels and feature history.

Slow-changing dimensions and the merge problem

A feature that changes rarely, a customer’s region, their account tier, their subscription plan, is a slow-changing dimension. The naive representation stores only the current value, which rewrites history and makes every past training row use today’s region. The correct representation stores versioned rows with validity intervals, as above, and appends a new row on each change rather than updating in place.

The failure this avoids is worth stating concretely. A region change caused by a company reorganization, applied as an in-place update, retroactively reassigns two years of training rows to the new region. A model trained after the update learns that region predicts the label, because the region now correlates with the post-reorganization outcome for exactly the rows that changed. The effect is a spurious time trend encoded in a categorical feature, and it is invisible in the data unless someone diffs the table across snapshots.

Streaming variants of the same join exist for online serving, where the lookup is a point query against a store whose rows carry a timestamp and the query returns the newest row at or before now. The consistency requirement between the batch and streaming versions is the same as described in training-serving skew: both must use the same interval convention and the same treatment of late arrivals.

Late arrivals at assembly time

The pipeline that builds the table has its own operational failure modes, and they produce rows that pass every timestamp check while being wrong. An event that arrives after the day’s partition has been written is either dropped or appended as a correction, depending on the warehouse. If it is dropped, the historical feature value for that entity on that day is lower than what a live system would have shown, and the model learns from a slightly deflated input. If it is appended, the daily partition no longer matches the snapshot the training job read, so a rebuild of the same training set produces different values unless the snapshot identifier is pinned.

The fix that is available in a mutable warehouse is to build the table from a snapshot that is declared complete, and to declare completeness only after the pipeline’s lateness bound has elapsed: the interval within which the source is contractually guaranteed to have delivered. Partitioned tables where each partition is annotated with whether it is final let the assembly skip incomplete days. Without that annotation, the only signal is age, which is a proxy for completeness rather than a guarantee.

Validation before the model sees the table

Two checks catch most violations cheaply. The first is a monotonicity check: for each feature, compute the correlation between its value and the label across the assembled table, and compare against the same correlation computed on a joined table built with a deliberately leaky join on latest values. A large gap is a red flag, but it is diagnostic rather than conclusive, since a genuinely predictive feature can be correlated too.

The second, sharper check is a future-event audit. For a random sample of labelled rows, ask whether any input event used to compute the feature has an event time or ingestion time later than the row’s anchor. This requires the pipeline to retain lineage from the feature back to the source rows, which is why the definition artifact described in feature definitions as versioned artifacts records its inputs by identity. Without lineage the audit cannot be performed at all, and the correctness of the table rests on review of the SQL.

A third check operates on the label side. Labels are defined by observation windows, a purchase within the next seven days, and the window’s end must be before the end of the training data snapshot. If the snapshot ends during a label’s observation window, the rows near the boundary have labels that are systematically wrong: their negative class contains events that had not happened yet when the data was extracted. Truncating the training window by the maximum label horizon fixes this, and the truncation must be applied per label definition rather than globally when several labels with different horizons share a table.

What the join cannot fix

Point-in-time correctness constrains when a value was computed, not what it depends on. A feature whose value is itself derived from a forward-looking aggregate, average order value over a customer’s lifetime, computed in a table that was only finalized last week, remains leaky no matter how carefully the join is anchored. The reconstruction of a historical feature value is only as honest as the historical snapshot of its inputs, which is why the training table’s fidelity depends on the warehouse’s retention: if the source table only retains ninety days, a training set spanning two years is necessarily assembled from current values and the leakage returns quietly.