Windowed Drift Detection and the Arithmetic of Its Alarms

A model is trained on a distribution and serves a distribution, and the two drift apart for reasons that have nothing to do with model quality: a new client version changes the input format, a marketing campaign shifts the user population, a supplier changes a sensor’s calibration. Detection is therefore a monitoring problem, and monitoring means comparing a sample of recent inputs against a reference sample and deciding whether the difference is larger than sampling noise.

The decision rule is a two-sample test. Its alarm rate under no real change is set by the test’s significance level, the number of comparisons made per period, and the correlation between consecutive windows. All three are under the control of whoever configures the monitor, and the usual configuration produces an alarm most days.

What is actually being compared

A drift monitor holds a reference distribution and a current window, and compares them. The reference is either the training set or a fixed slice of production traffic from a known-good period. The choice matters: a reference taken from the training set conflates drift with the gap between offline data and online data, which exists at launch and never goes away.

The current window is a sample of feature values from recent requests. Its length trades two errors. A short window reacts quickly and has high variance, so small shifts are invisible against noise. A long window has low variance and smears a step change across the whole window, so the alarm arrives late and weak.

The interaction with traffic volume is what makes a fixed window length wrong. A window defined as one hour contains a different number of observations at 3 a.m. than at noon, and the test’s power changes with the count. A window defined as a fixed number of observations has constant power and variable latency, which is usually the better trade because the alarm’s meaning stays stable.

Test choice and what each assumes

Three families are used, and they fail differently.

Population stability index bins both samples into the same buckets, computes the share of each sample in each bucket, and sums $(\text{current share} - \text{reference share}) \times \ln(\text{current share} / \text{reference share})$. The sum is a symmetrised divergence with no sampling distribution attached. The conventional thresholds of 0.1 and 0.25 are rules of thumb, not significance levels, and they do not account for sample size: the same shift produces a larger index when the bins are fine and the samples are small, because empty bins produce infinite terms. The usual patch is to add a small constant to every share, which caps the contribution and makes the index depend on the constant.

Chi-square goodness of fit compares observed counts per bin against expected counts scaled from the reference. It has a known null distribution, so a p-value is available, and it requires expected counts of roughly five or more per bin. With thirty bins and a window of a hundred observations, that condition fails and the test’s nominal level is wrong. The bin count therefore has to be chosen from the window size, not from a desire for resolution.

Kolmogorov-Smirnov compares empirical cumulative distributions and applies to continuous features without binning. Its null distribution is distribution-free, which is its main attraction, and it is most sensitive near the middle of the distribution, which is where drift usually does not matter. A shift confined to the upper tail, which is exactly the case for latency and for many transaction amounts, produces a small KS statistic.

None of the three detects a change in the conditional relationship between features and labels. They compare marginals, one feature at a time. A model can be badly miscalibrated while every marginal is unchanged, because the joint distribution moved. Detecting that requires monitoring outcomes, which requires labels, which arrive late.

The multiple-comparison problem in a feature table

A model with two hundred features, each monitored with a test at the 1% level, produces two false alarms per window on average when nothing has changed. At a window of one hour, that is roughly fifty false alarms per day. The monitor is then either ignored or tuned until it stops firing, and both outcomes lose the signal.

The arithmetic is direct: with $m$ independent tests at level $\alpha$, the expected number of false positives per period is $m\alpha$. Independence is optimistic because features are correlated, which reduces the effective count, but the reduction is not something to rely on without measuring it.

Three approaches reduce the rate.

Bonferroni correction divides the per-test level by $m$, so each feature is tested at $\alpha/m$. It controls the family-wise error rate, meaning the probability of any false alarm in a window, at the cost of power: a genuine shift in one feature must be large to clear a threshold divided by two hundred. For drift detection, where a real shift often affects a correlated group of features, this is harsher than necessary.

Benjamini-Hochberg controls the false discovery rate instead: the expected fraction of flagged features that are false. Sorting the $m$ p-values and comparing the $i$-th smallest against $i\alpha/m$ gives a threshold that adapts to how many features are actually shifted. When one feature moves strongly, its p-value is far below its threshold and it is flagged; the others stay quiet.

Grouping is the third approach and often the most useful. Features are correlated in blocks, so a test on a block-level summary, such as the mean of a group of related counts, is one test rather than twenty. The number of tests falls, the power per test rises because the summary aggregates more data, and the alarm is more interpretable because it names a group rather than a column.

Windows overlap, so alarms are not independent

Consecutive windows almost always overlap, either because the monitor uses a sliding window or because the metric is computed on a rolling basis. Overlapping windows share observations, so their statistics are positively correlated, and the effective number of independent tests per day is smaller than the count of windows.

The practical consequence is a suppression of the alarm rate relative to the independent approximation, which sounds helpful and is not: it also means a shift that persists is repeatedly detected by overlapping windows, producing a burst of alarms for one event rather than one alarm. Any alerting layer on top needs to deduplicate, usually by requiring the condition to hold across a minimum number of consecutive windows or by suppressing after the first alarm for a cooldown period.

The other correlation is across features within a window. A shift in a shared upstream cause moves several features together, which makes the tests dependent and makes the effective $m$ smaller than the column count. Estimating the effective count from the correlation matrix is possible and rarely done; grouping by known lineage is the practical substitute.

Choosing a reference that does not drift with the detector

A reference computed once from training data is stable and becomes stale as the legitimate distribution evolves. A reference recomputed from recent traffic is current and drifts with the thing being measured, so a slow shift is never detected: each window is compared against a reference that has already moved.

The middle path is a reference that is refreshed deliberately, on a schedule or after a confirmed change, and pinned by version so that the comparison is reproducible. A reference refresh is a change to the detector, and it should be recorded as one, because the alarm rate before and after is not comparable.

There is a subtler trap in refreshing from recent traffic: if the drift was caused by a defect, such as a client sending a malformed field that the pipeline coerces to a default, the refreshed reference absorbs the defect and the monitor goes quiet. Refreshing only after a confirmed legitimate change, and never automatically on a schedule alone, prevents the detector from learning the bug.

Detection latency and the cost of the delay

A drift detector’s value is bounded by how quickly it fires relative to how quickly the harm accumulates. For a feature that feeds a ranking model, a day of undetected shift is a day of degraded rankings. For a feature that feeds a fraud model, the same delay is a day of losses.

Detection latency is roughly the window length plus the number of windows required for the test to accumulate enough evidence. A window of ten thousand observations on a service receiving a hundred requests per second fills in about two minutes, so the latency is minutes. On a service receiving one request per second, the same window takes nearly three hours.

The consequence is that a single window size cannot serve both a high-traffic and a low-traffic feature in the same model. The low-traffic feature needs either a longer window, which delays detection, or a coarser summary, which reduces power. Making that choice explicitly, per feature group, is the difference between a monitor that fires usefully and one that fires constantly on the high-traffic features and never on the rest.

The monitoring described here detects that inputs changed. It does not determine whether the change harmed anything, and the two questions are answered by different mechanisms: harm is measured by outcomes, either through the replay machinery in Logged-Traffic Replay or through the guardrails described in Canary Promotion as a Repeated Statistical Test. A drift alarm is a prompt to ask the second question, not an answer to it.