Detecting Distribution Drift in Production Data

A model trained on last quarter decays. The decay may come from the world changing, from a broken upstream table, or from a new client version sending different fields. All three appear as the same symptom: the distribution of what the model receives no longer matches what it learned. Detecting that condition early is a statistical problem with a specific difficulty, modern feature vectors have thousands of dimensions, most of them discrete, and the number of tests performed per model per day is large enough that a naive per-feature alert produces mostly noise.

Windows, references, and the comparison being made

A drift check compares two samples. The reference is a fixed window of data, usually the training set or the first weeks after deployment. The current sample is a recent window, most often one day or one hour, chosen to match the cadence at which the system could act on an alert.

The choice of reference determines what the test means. Comparing against the training set detects any departure from the fitted distribution, including slow legitimate trends, and will eventually fire on every model that lives long enough. Comparing against a trailing window detects abrupt change but not gradual drift, because the reference moves with the data. The two answer different questions, and a system usually needs both: a fixed reference with a loose threshold for the slow signal, and a rolling reference with a tight threshold for step changes.

Sample size bounds the sensitivity. A two-sample test over a window of $n$ observations cannot reliably detect differences smaller than roughly $O(n^{-1/2})$ in distributional distance, so a daily window over a million requests detects shifts far smaller than a window over a thousand. The same property means that with a large enough window, every test rejects: the null hypothesis of identical distributions is false in any real system, because no two days are identical. Statistical significance is therefore not the useful output; the useful output is an effect size with a threshold chosen from operational tolerance.

Choosing a statistic per feature type

Numeric features can be compared with the Kolmogorov–Smirnov statistic, which is the maximum absolute difference between the two empirical cumulative distribution functions. It is sensitive to any difference in location, spread, or shape, requires no binning, and has a distribution-free null distribution that depends only on sample sizes. Its weakness is the same as its strength: because it takes a maximum over the whole support, a difference confined to a rare tail is weighted equally with one at the median, and it is insensitive in exactly the region where a skewed feature carries the most information.

The population stability index, a binned variant, is the practical alternative in monitoring systems built around fixed bins. It discretizes the reference into quantile bins, then sums $\sum_i (p_i - q_i)\log(p_i/q_i)$ over bins, where $p$ and $q$ are the reference and current proportions. The logarithm makes it unbounded above and unbounded below only when a current bin is empty, so implementations add a small constant to empty bins. Its value depends on the binning, which makes cross-model comparison meaningless but within-model tracking useful: a stable model’s index stays near zero, and the threshold is a policy choice rather than a p-value.

Categorical features with many levels resist both. A chi-squared test over a contingency table is invalid when many cells have small expected counts, which is the normal case for a category with thousands of levels. The workable approach is to compare the frequency vector under a distance that handles sparsity, total variation over the top levels plus a collapsed “other” bucket, or the Jensen–Shannon divergence, which is bounded and symmetric. Rare categories should be monitored collectively rather than individually, because the per-category rates on a rare level are too noisy to threshold.

Missingness deserves its own test. A change in the null rate of a column is often the first visible symptom of an upstream schema change, and it shows up even when the non-null values are unchanged. A simple proportion comparison between windows, with a threshold on the absolute difference, catches it before the downstream distribution tests react.

Detecting shift in the joint distribution

Per-feature tests miss shifts that leave each marginal intact and change the correlation structure, a client that sends two fields swapped, or a join that now pairs features from different entities. Multivariate tests address this by reducing to a one-dimensional statistic.

The classifier two-sample test trains a classifier to distinguish reference rows from current rows. If the two samples come from the same distribution, the classifier’s held-out accuracy is at chance; above-chance accuracy measures how distinguishable they are, in a metric that is directly interpretable and that integrates over all features. The cost is that it requires training a model to monitor a model, and the diagnostic is weak: the classifier says the distributions differ, not which features caused it, unless feature importances are inspected afterwards.

Kernel two-sample tests, the maximum mean discrepancy with a Gaussian or Laplacian kernel, give a consistent test with a computable null distribution, at $O(n^2)$ cost in the sample size for the exact statistic. Subsampling the window into batches of a few thousand rows makes the cost manageable, and the loss of power from subsampling is small relative to the gain from catching joint shifts.

A cheaper proxy is to monitor the model’s own outputs. The distribution of predicted scores, or of a fixed set of decisions derived from them, is a single one-dimensional signal that responds to shifts in any input that the model weights heavily. A shift in the score distribution does not identify the cause, but it is nearly free to compute and it fires for shifts that matter to the business. Its blind spot is a shift along a direction the model ignores, which is precisely the case where the model’s behaviour is unchanged and the alarm would be academic.

Thresholds that survive a thousand tests

With one test per feature per hour and a thousand features, a fixed per-test false positive rate of one percent yields ten alerts an hour by construction. Three practices keep the alert volume usable.

Alerts are computed on effect size, not p-value, with the threshold set from the tolerance of the downstream decision rather than from statistical convention. A feature whose shift does not move any decision is not worth an alert, and the metric that encodes that is the change in the model’s predictions if the feature were resampled from the reference distribution.

Tests are aggregated rather than reported individually. The maximum of a set of statistics, with its null distribution estimated by permutation over the reference window, controls the family-wise error rate across features without a Bonferroni correction that would be far too conservative given the correlation between features. The permutation approach also handles the fact that features derived from the same source are strongly dependent, which breaks the independence assumption behind most multiple-testing corrections.

Alerts are conditioned on preceding signals. A sudden change in a feature that coincides with a deployment of an upstream service is a different event from the same change with no deployment, and the monitoring system’s value comes partly from joining the two streams. Without that join, the on-call engineer performs it manually, which is the step that fails at three in the morning.

What no monitoring detects is a shift that is real, gradual, and unpredicted by any input statistic until it has already degraded outcomes. That case is caught by watching the label, with the delay the label’s horizon imposes, and it is the reason drift alarms are a leading indicator rather than a substitute for outcome monitoring. How a validated input contract fails in its own characteristic ways is treated in data validation at pipeline boundaries, and the feedback loop that turns observed outcomes into a retraining decision is the subject of closing the retraining loop.