Seasonal Baselines and the False Alerts They Prevent

A pipeline that processes several times as many records on a weekday as on a Saturday has a volume metric with a strong weekly cycle. A check that alerts when volume falls more than thirty percent below the trailing average fires every Saturday, and a weekly false alert is enough to get the check muted. The same threshold would also have flagged a genuine partial load that cut a Tuesday’s volume by a third, in the period before it was silenced.

The threshold was not wrong; the baseline was. Comparing a value to an average that mixes weekdays and weekends measures the day of the week, not the health of the pipeline. Fixing it requires a baseline that carries the same periodicity as the metric, which is a different computation from a moving average.

What a baseline has to reproduce

A baseline is a prediction of what a value should be, and the check is a comparison of the observed value against it. The baseline’s job is to be wrong only when something has changed, so it has to absorb every source of variation that is not a fault.

Three sources dominate in operational metrics. The first is trend: a dataset that grows five percent a month has a volume that rises monotonically, and a baseline that does not track the trend will drift out of range within a few months. The second is periodicity: daily, weekly, and annual cycles, with the weekly cycle usually the strongest for business data and the daily cycle strongest for infrastructure metrics. The third is the irregular component, which is what a check is trying to detect.

A moving average over $k$ periods absorbs trend only if $k$ is short relative to the trend’s timescale, and absorbs periodicity only if $k$ is a multiple of the period. A seven-day moving average over daily data removes the weekly cycle exactly, because every window contains one of each weekday. A fourteen-day window does the same. A ten-day window does not, and it produces a baseline whose value depends on which weekdays happened to fall in the window, which means the baseline itself oscillates with a period that has nothing to do with the data.

That arithmetic is the whole argument for period-aligned windows, and it is easy to get wrong in a system where the window is configured in days and the period is not a whole number of days. A monthly cycle is 30.44 days on average and 28 to 31 in practice, so no fixed window removes it.

Same-period comparison, and its blind spot

The simplest seasonal baseline compares a value to the same period one cycle earlier: Tuesday to last Tuesday, this week to last week, this month to the same month last year.

Its virtue is that it requires no model. Its weakness is that it has one observation per period, so it cannot distinguish a real deviation from ordinary noise. If Tuesday volumes naturally vary by ten percent, a comparison to a single previous Tuesday will flag any deviation beyond that, including the many that are simply the low end of the natural range. The result is an alert rate that is proportional to the noise, and noisy metrics produce alert rates that make the check useless.

The remedy is to compare against several cycles at once: the same weekday over the last four to eight weeks. That gives a sample from which a centre and a spread can be estimated, and the spread is what sets the threshold. The centre is the seasonal expectation, and the spread is the metric’s natural variability within that season.

Using several cycles introduces a lag in adaptation. A genuine step change in the metric will be averaged into the baseline over the number of cycles in the window, so detection takes that many periods. With four weeks of history, a permanent shift takes about four weeks to be absorbed, during which the check fires every period. That repeated firing is the correct behaviour for a real change and the wrong behaviour for a transient one, and distinguishing them requires looking at the persistence of the deviation rather than a single period’s value.

The threshold has to be relative to the spread

A threshold expressed as a fixed percentage of the baseline assumes the metric’s variability is a fixed percentage of its level, which is true for some metrics and false for others.

A count of events that follows a Poisson-like process has a standard deviation proportional to the square root of its mean, so the relative variability shrinks as the volume grows. A metric with a million events has a relative standard deviation around a tenth of a percent; one with ten thousand has around one percent. A fixed ten percent threshold is therefore far too loose for the large metric and far too tight for the small one, and no single percentage works across a table’s partitions, which routinely differ in volume by orders of magnitude.

A metric whose variability comes from a mixture of sources, such as a total that sums a heavy-tailed distribution, has variability that scales with the level rather than its square root. For that metric a percentage threshold is closer to right, though the tail means the distribution of the metric is skewed and a symmetric band will fire more often on the high side.

The answer that tolerates outliers is to estimate the spread from the same historical window used for the centre, and to set the band in units of that spread. The median absolute deviation is the usual estimator, because it is insensitive to the very outliers the check is meant to detect: a window containing one catastrophic day will have a mean dragged by it and a median absolute deviation barely moved. The scaling constant that converts median absolute deviation to a standard-deviation equivalent for normal data is about 1.4826.

from statistics import median

def seasonal_baseline(history_by_weekday, weekday, k=3.0):
    """Centre and band for one weekday, from that weekday's past values."""
    sample = sorted(history_by_weekday[weekday])
    if len(sample) < 3:
        return None                      # too few cycles to estimate a spread
    centre = median(sample)
    mad = median(abs(x - centre) for x in sample)
    spread = 1.4826 * mad
    # Floor the band so a perfectly stable history does not produce a zero-width threshold.
    spread = max(spread, 0.02 * centre)
    return centre, centre - k * spread, centre + k * spread

Why alert volume is a design parameter

Every check has a false-positive rate, and the rate determines whether the check survives. A check that fires once a week is investigated. A check that fires twenty times a day is muted, and a muted check has a detection rate of zero regardless of its sensitivity.

The arithmetic is unforgiving. A check applied to a thousand partitions, each with a five percent false-positive rate per period, produces fifty alerts per period. At that volume the alerts are noise, and the operator’s response is to raise the threshold until the volume is manageable, which lowers the sensitivity of every partition to accommodate the worst-behaved ones. The aggregate behaviour of the system is set by the least stable partition, and the stable partitions pay for it.

Two structural remedies exist. The first is to calibrate per entity rather than globally, so each partition’s band reflects its own variability; a stable partition then gets a tight band and a volatile one gets a loose band, and both have the same false-positive rate. The second is to alert on aggregate state rather than on individual events: a partition that deviates once is recorded, and an alert fires only when the count of deviating partitions in a period exceeds what the per-partition rates would produce by chance.

The second has more statistical power and is harder to reason about, because it requires modelling the distribution of the count rather than of the metric. It is also the only approach that scales to estates where the number of entities is large enough that some entity is always deviating.

Deviations that are not faults, and faults that are not deviations

The correspondence between a statistical deviation and a defect is not one-to-one, in either direction.

A deviation without a defect: a marketing campaign raises traffic, a public holiday lowers it, a downstream consumer stops querying and a cache stops warming. These are real changes in the metric that are not failures of the pipeline. A detection system that cannot distinguish them will produce alerts that are technically correct and operationally useless, and the only durable remedy is a way to declare known events so the check can account for them. The declaration is a piece of metadata with a validity interval, and it is the anomaly-detection equivalent of a maintenance window.

A defect without a deviation: a transform that swaps two columns produces a volume that is exactly as expected, and a join that matches the wrong rows but the same number of them does the same. These are the failures that volume checks cannot see at all, and they are why the detection layer needs content signals in addition to volume. The null rate and cardinality checks cover part of this space, and relational assertions cover the rest.

The asymmetry is worth stating plainly: statistical detection finds changes, and defects are a subset of changes. Everything a detector can do is bounded by that subset, which is why a detection layer is never a substitute for assertions that encode what correct means.

Baselines have to be maintained, or they rot

A baseline computed from history is a derived object, and it decays. Three decay modes matter.

The first is the schema change. A column added upstream changes the metric’s level, and the historical window now spans two regimes. Until the window rolls past the change, the baseline is a blend of both, and its spread is inflated by the step. The check is insensitive during that period, which is exactly when a fault is most likely, because the change itself is the risk.

The second is the definition change. A metric whose SQL is edited to exclude a category has a level change that is indistinguishable, from the data alone, from a fault. Recording the metric’s definition alongside its baseline, and invalidating the baseline when the definition changes, is the only way to keep the two separate.

The third is the population change. A baseline for a partition keyed by region assumes the set of regions is stable. When a region is added, its partition has no history and the baseline is undefined; when a region is removed, its baseline persists and its absence looks like a drop to zero. Handling both requires the check to be aware of the expected population, which is a metadata question rather than a statistical one.

The maintenance burden is the reason many detection systems converge on a simple, explainable baseline rather than a fitted model. A seasonal decomposition with a fitted trend can absorb more variation, and it also has more parameters that can be wrong, more ways to be stale, and a failure mode that is harder to explain to the person receiving the alert. The comparison of an observed value against a modelled expectation has the same shape in the freshness domain: a simple quantile of the age distribution is easier to defend than a fitted latency model, and it fails in ways that are visible.