Training-Serving Skew and the Contract Between Features and Models

A model is a function fitted to numbers, but the numbers it is fitted to are produced by code that was never part of the model. That separation is where the first serious systems problem in machine learning appears. The distribution of the values a model sees at inference time can differ from the distribution it was fitted on, not because the world changed, but because a different program computed them. The training pipeline reads a table of historical events and joins them; the serving path receives one request and looks up whatever it can find within a latency budget. Both are called “the feature”, and they are not the same function.

One feature, two implementations

Consider the ordinary case of a purchase-rate feature for a recommendation model: the number of orders a user placed in the last thirty days. In training it is a SQL aggregate over an append-only orders table:

SELECT user_id,
       COUNT(*) FILTER (WHERE placed_at > as_of - INTERVAL '30 days') AS orders_30d
FROM orders
WHERE placed_at <= as_of
GROUP BY user_id, as_of;

The as_of column is the point-in-time anchor: for each training row, the aggregate is computed using only events that existed before the label was observed. This matters because the label for a given row usually refers to behaviour after as_of. Using the full history would leak the future into the feature, and the resulting model would look excellent offline and fail immediately in service.

In serving, the same feature is computed by a service that queries a time-series store or a cache populated by a stream. It increments a counter when an order event arrives, decrements or re-buckets when the event ages past thirty days, and returns the current value. That implementation is not a translation of the SQL. It is a different program: different data source, different staleness profile, different treatment of late-arriving events, different handling of users with no orders at all.

Three specific divergences tend to appear:

• Time zone. The batch path groups by a calendar day in one zone; the stream path buckets by epoch milliseconds. Events near midnight fall on different sides of a boundary.
• Missing data. The SQL GROUP BY simply produces no row for a user with no orders, and the join leaves a null that the pipeline fills with zero. The serving counter may return zero or an error depending on whether the key exists in the cache, and the error might be swallowed by a fallback default that was never applied in training.
• Deduplication. The batch source is re-delivered in full each day and can be deduplicated by primary key. A streaming counter sees each event once if the stream is exactly-once and twice if it is at-least-once, which is the common case for a queue that acknowledges after processing.

Each of these changes the numeric input to the model for a small fraction of requests. The model’s coefficients were fitted under the assumption that a value of 3 means what it meant in the training table. When one percent of requests arrive with a value inflated by a duplicate, the model does not know; it applies the coefficient it learned.

Why the split is structural

The naive fix is to compute the feature in one place and use that code in both paths. This fails for reasons that are properties of the two environments, not of the engineering culture.

Training reads history, so it can afford to scan a year of events for each row, join across tables in whatever order is convenient, and take minutes per batch. Serving reads the present for one entity, and the whole request budget, feature computation included, is measured in tens of milliseconds. Training runs over a fixed snapshot, so its results are deterministic and reproducible. Serving runs against live stores whose contents change between the first and last feature lookup of a single request.

The two backends also differ in what they can express. Batch SQL has joins, window functions, and no restriction on how far back it may look. A low-latency key-value store has point lookups and range scans. A feature that is a thirty-day count is easy in both. A feature that is “average order value across the user’s three most similar neighbours” is easy in batch and requires an approximate nearest-neighbour index in serving, with a different and approximate answer.

The result is that most real systems own two implementations, and the engineering question is how to keep them agreeing, not how to avoid having two. The common architecture that accepts the split is a feature store: a system that computes features in a batch job, writes the results to an offline store for training and to an online store for serving, and thereby makes both paths read the same values rather than recompute them. That solves skew for the features it covers entirely, at the cost of a freshness delay between the event and its availability online, plus a second copy of every feature that must be kept in sync. Streaming-only designs compute one implementation and backfill it for training, which removes the divergence but requires the stream to support replay over the full history, and requires the same exactly-once guarantees at replay time that it offers live.

Detecting divergence without a live comparison

The direct test is to log the feature vector at serving time and compare it against the batch computation for the same entity and timestamp. This is expensive per request and usually done on a sample. Even at low sample rates it catches the important cases, because skew like a time-zone mistake or a units change affects a large fraction of rows rather than one.

A cheaper structural test is to run both implementations over the same input. The batch pipeline produces a table of (entity, timestamp, feature vector); the serving implementation, invoked as a function over the same rows, produces another. Comparing the two column by column over a day of traffic gives a divergence rate per feature. A tolerance is unavoidable because floating-point accumulation order differs, but the tolerance can be tight for counts and ranks and looser for sums over floating-point events.

Two further checks operate on aggregate statistics rather than individual values. First, the mean and standard deviation of each feature over the last day of serving traffic can be compared with those of the most recent training window; a shift of several standard deviations in the input of a linear model moves the output proportionally. Second, the model’s output distribution itself can be tracked: an unchanged model receiving shifted features produces a shifted score distribution, and that shows up in a histogram before it shows up in a business metric. The mechanisms for turning such comparisons into alerts are the subject of detecting distribution drift in production data, which treats the statistical side in detail.

Neither check distinguishes skew from genuine change in the world. A model whose inputs shift because user behaviour changed and a model whose inputs shift because a join broke produce the same signal at the aggregate level. The distinction comes from the paired comparison over identical rows: if the two implementations agree on the same input, the shift is real data; if they disagree, the shift is software. That property makes the paired test the more valuable of the two, and it is why the comparison harness is worth building before the model is trained rather than after it misbehaves.

Leakage as skew’s mirror image

The same two-path separation produces the opposite error in training. A feature computed with information that would not be available at serving time is leakage, and it is structural for the same reason skew is: the batch path has access to the full table, including columns that only exist because the future already happened. The as_of predicate in the SQL above is the guard, and it is easy to omit. When it is omitted, the model learns from a feature whose distribution during training includes post-outcome information, and its offline metric is inflated by an amount that depends on how predictive that leaked column is.

Leakage and skew therefore form a single discipline: every feature must be expressible at a specified point in time using only events before that point, and the training computation must enforce the cut while the serving computation must be able to reproduce it for a single entity. A feature that cannot satisfy both constraints is not ready to be used, regardless of how well it correlates with the label on the training window. The bookkeeping that makes this auditable, what each feature depends on, when each was last recomputed, which model version consumed which feature version, is what the rest of this cluster builds on, starting with the definition of a feature itself in feature definitions as versioned artifacts.