Seasonality and the Limits of Static Alert Thresholds

A threshold is a horizontal line, and most quantities worth alerting on do not move horizontally. Request volume, queue depth, and connection count follow daily and weekly cycles whose amplitude often exceeds the size of the excursion that matters. A fixed threshold placed above the daily peak fires only on events larger than the peak itself, which means ordinary load is enough to trigger it. A threshold placed at the average fires every day, twice.

The observation is not an argument against thresholds. It is an argument for choosing which quantity the threshold is applied to. Two of the three responses to seasonality reduce the problem rather than modelling it: alert on a ratio whose numerator and denominator scale together, or alert on a quantity that has been transformed to remove the periodic component. The third, fitting a model of the seasonal pattern, is the most flexible and the most expensive to maintain.

Ratios that scale with load

The burn rate is the first response and it explains why availability alerting is largely immune to seasonality. The indicator is a ratio of failing to total requests, and both parts scale with traffic, so the ratio’s expected value under normal operation is flat across the day. A service whose error ratio is stable at 0.05 percent at both the nightly trough and the morning peak produces the same burn rate at both, which is the property that lets one threshold cover the whole cycle.

The immunity is partial, and the exceptions are where seasonal alerting is actually needed. Some failure rates are correlated with load: connection pool exhaustion, cache misses following a cold start, and timeouts on a dependency that also has a daily cycle all produce a higher error ratio at peak. A threshold set to catch the peak condition fires at every peak. The correct remedy is usually to model the correlation rather than to widen the threshold, because widening it to cover the normal peak removes the ability to detect a moderate degradation at the trough.

Latency objectives have a related but distinct problem. A latency distribution shifts with load, so a fixed latency threshold corresponds to a different percentile at different times of day. An objective stated as “the fraction of requests faster than 300 milliseconds” is a ratio and behaves like the error ratio, which is why the objective should be stated that way rather than as a percentile threshold. Where the underlying quantity must be alerted on directly, its seasonal component has to be removed.

Removing the cycle by comparison

The cheapest way to remove a cycle is to compare the value with itself at a lag of exactly one period. For a daily cycle, the current value is compared with the value twenty-four hours earlier; for a weekly cycle, with the value seven days earlier. The difference or ratio of the two has no daily component if the pattern is stable, and a threshold on it can be tight.

from statistics import median

def same_time_baseline(series, period=288):
    """series: 5-minute samples. period=288 is one day.
    Returns the lagged value and an outlier-resistant spread of recent changes."""
    lagged = [v for v in series[:-period] if v is not None]
    deltas = [abs(series[i] - series[i - period])
              for i in range(period, len(series))
              if series[i] is not None and series[i - period] is not None]
    return series[-period], median(deltas) * 1.4826  # MAD scaled for a normal

The scaled median absolute deviation is the outlier-resistant alternative to the standard deviation: multiplying the median absolute deviation by roughly 1.4826 makes it estimate the standard deviation of a normal distribution, while remaining insensitive to the outliers that a standard deviation is dominated by. For alerting, insensitivity to outliers matters in both directions. A single spike inflates a standard deviation enough to hide the next genuine excursion, and a baseline computed from a mean is dragged by any sustained anomaly that occurred during the baseline window.

The lag has three failure modes that are not obvious from the construction. The first is that it assumes the period is exactly known and stable. A daily cycle whose peak shifts by an hour across the year produces a lagged comparison that is systematically biased during the transitions, which in a service with a pronounced peak produces false positives twice a year. The second is that the lagged value must itself be trustworthy. If the load generator that produces the comparison value was running at the time and the service then changed, the baseline is wrong by the size of the change. The third is a cold start: a newly deployed service has no value from one period ago, so the rule either has no data or uses whatever early samples exist, both of which are unreliable.

Fitted seasonal models

Where the lag comparison is too coarse, the periodic component can be estimated directly. The classical construction decomposes a series into a trend, a seasonal component, and a residual, and alerts on the residual. Simple exponential smoothing tracks a level; double exponential smoothing adds a trend; triple exponential smoothing, usually named after Holt and Winters, adds a multiplicative or additive seasonal term with one coefficient per position in the cycle. The alert threshold then applies to the residual, whose spread is much smaller than the original series.

The cost of fitting is the maintenance of the model, and the failure modes are all consequences of fitting. A model trained during a period of degraded performance encodes the degradation as normal and never alerts on it; this is the baseline-drift problem, and it is why the baseline window must be long enough to span a full cycle but short enough that a month-old regression has not been absorbed. The model has no way to distinguish a regression that has been present for the whole window from correct operation, so the window’s length is effectively the detection horizon for slow regressions.

Fitted models also have an edge behaviour. At the boundary between the baseline window and the present, the model’s parameters are estimated from data that includes the ongoing condition, so an anomaly that begins exactly at the boundary is partially absorbed. Overlapping windows and refitting on a schedule adjust the size of this effect but never remove it.

Differencing and the noise it creates

Alerting on a rate of change is the third common response, and it has a quantitative cost that is easy to state and easy to forget. If a series has independent noise with variance $\sigma^2$, the $h$-step difference has variance $2\sigma^2(1-\rho)$ where $\rho$ is the lag-$h$ autocorrelation. For a series with no autocorrelation at that lag, the variance doubles, so the standard deviation rises by a factor of about 1.41. Computing a rate by dividing the difference by the interval $h$ amplifies that further by $1/h$, so a derivative estimated from short steps is dominated by noise and a derivative estimated from long steps lags the event it is meant to catch.

There is a real use for derivative alerts: a monotonically growing quantity such as disk usage or a memory leak has a slope that is stable and meaningful even when its value is noisy. The slope of a counter over a long window is exactly what a rate function computes, and comparing that slope with the trend of the trend is the basis of an exhaustion forecast, which Forecasting Exhaustion for Saturation Alerts treats as a distinct rule shape. The mistake is to apply a derivative to a quantity whose value is already the signal, which replaces a well-behaved series with a noisier version of itself.

Selecting the mechanism honestly

The three responses have different requirements and different failure surfaces. A ratio that scales with load needs the load to scale the numerator and denominator together, and it fails when the failure mechanism is load-dependent. A lagged comparison needs a stable period and enough history, and it fails at period boundaries. A fitted model needs a maintained baseline and fails by absorbing regressions.

The choice should be driven by what the rule is for. Availability and latency objectives are ratios and are best alerted on with burn-rate rules, which do not need a seasonal model because the ratio is self-normalising, as the derivation in Error Budgets and the Burn Rate That Consumes Them shows explicitly. Resource saturation, capacity headroom, and throughput anomalies are not ratios and do need one of the other two mechanisms. Reaching for a learned model where a ratio would do is a common and expensive mistake, because the model adds a component that can itself fail, and its failures are harder to diagnose than a threshold that was set too low.