Alert State, Hysteresis, and the Cost of Flapping

An alert rule is evaluated on a schedule, and each evaluation produces a set of label sets rather than a single boolean. A rule that matches nothing returns an empty result, which is not the same as returning zero; a rule that matches a single series returns one vector element. The state machine consumes that result and moves each identified alert instance through discrete states, and almost every oddity in alerting behaviour follows from the interaction between the evaluation schedule and those states.

The state machine and what drives it

An instance that the rule’s expression does not currently select is inactive. When an evaluation selects it, the instance becomes pending, and it stays pending while successive evaluations keep selecting it. Once the condition has held for the configured hold duration, conventionally written as a for clause, the instance becomes firing. If an evaluation stops selecting it before the hold duration is satisfied, the instance returns to inactive without ever having fired, and no notification is produced.

Firing has one more transition. When an evaluation stops selecting a firing instance, it moves to inactive and a resolution is emitted. Implementations that offer a retention field for firing state hold the instance in firing for a configured period after the condition clears, which exists precisely for the oscillating case described below.

Two properties of this machine cause most of the confusion.

The first is that the hold duration is measured in evaluations, not in wall-clock seconds relative to the condition. A rule with a for of five minutes, evaluated once a minute, requires the condition to be selected by six consecutive evaluations: the one that first selects it plus five more. The observed delay from onset to firing is therefore between the hold duration and the hold duration plus one evaluation interval, and it is further offset by however much of the condition was present before the first evaluation that saw it.

The second is that alert identity is the label set. If any label changes while the condition holds, the state machine sees a different instance: the old one clears and a new one starts pending from zero. A rule whose expression includes a pod name will reset its own pending timer every time a pod is replaced, and a rolling restart of a deployment can therefore prevent the alert from ever firing. Labels used for identity should be the ones that are stable across the incident, which is the same set the router groups on.

Why a threshold at the boundary oscillates

A condition that hovers near its threshold crosses it repeatedly under ordinary variation. Success ratio near a target, queue depth near a capacity limit, and latency near a percentile goal all behave this way, because the underlying quantities are noisy and the threshold is a single line through the middle of the noise.

The consequence for a rule without a hold duration is a sequence of firing and resolved events at the cadence of the evaluation interval. Each transition can produce a notification, and the notification stream then has a volume proportional to the number of threshold crossings rather than to the number of incidents. The page count for one underlying condition can reach the tens per hour by this mechanism alone, and the responder’s experience is a channel that repeats the same message with alternating wording.

Grouping and repeat intervals absorb part of it, through the router timers described in Routing Alerts by Required Response. A router that groups by the alert’s identity and applies a repeat interval will collapse a burst of transitions into a smaller number of notifications, and the state at any moment is still visible in the router’s view. But grouping suppresses notifications, not transitions, and the router’s own grouping timers interact with the rule’s evaluation cadence in ways that are hard to predict. The alert volume reaching the responder ends up depending on two independent timing configurations, neither of which is documented next to the other.

What a hold duration actually buys

The for clause is the first defence. Requiring the condition to hold for five or ten minutes removes any oscillation whose period is shorter than the hold duration, because each excursion past the threshold fails to reach the firing state. The cost is the detection delay. A condition that holds continuously is delayed by up to the hold duration plus one evaluation interval, which is the same for every alert on that rule, including the severe ones.

The asymmetry matters for choosing the value. Hold durations are usually applied per rule, so a rule that pages cannot have a hold duration long enough to suppress a ten-minute oscillation without also delaying by ten minutes the condition it was built to catch. That is why two-window burn-rate rules are preferable for symptoms: the short window performs the filtering function in continuous time and resets faster than a hold duration would, as described in Multiwindow Burn-Rate Alerts and the Short Window as a Reset. The for clause then only has to cover the case where both windows cross and fall back within one evaluation.

A hold duration has a second effect that is easy to overlook: it filters conditions by their duration, not by their depth. A burn rate far above the threshold that lasts four minutes and a burn rate barely above the threshold that lasts ten minutes are treated identically by a five-minute hold. If the two cases warrant different responses, which they usually do, the hold duration is not the tool; separate rules at different thresholds are.

Hysteresis as a second threshold

Hysteresis removes the oscillation at its source by making the condition for entering the firing state different from the condition for leaving it. The classic form uses two thresholds: fire when the value exceeds the upper one, clear when it falls below the lower one, and hold the current state for values between them. The gap is the hysteresis band, and it is sized to exceed the amplitude of ordinary variation.

# Hysteresis for a ratio-style alert: enter only above `high`,
# leave only below `low`, hold state otherwise.
def step(value, firing, high=0.05, low=0.02):
    if firing:
        return value >= low        # stay firing until clearly recovered
    return value > high            # enter only on a clear excursion

The same shape appears in the two-window construction, where the “state” is the alert’s own firing status and the two windows play the roles of the two thresholds. In a rule that fires on the conjunction of a long and a short window and clears when either fails, the short window acts as the lower threshold: the alert stops firing as soon as the current state is healthy, regardless of how bad the accumulated window looks. That is hysteresis expressed in time rather than in value, and it produces the same drop in transition count.

The cost of hysteresis is a blind region. While the value sits inside the band, the alert’s state is whatever it was, so a condition that degrades from healthy into the band and stops there is never reported. The band must therefore be narrow enough that conditions inside it are genuinely uninteresting, and the value inside it must still be visible on a dashboard. A hysteresis band chosen wide enough to silence a noisy alert can hide a permanent regression of nearly the band’s width.

Measuring the oscillation before choosing a fix

The value of any of these mechanisms depends on the period and amplitude of the oscillation, which are properties of the series rather than of the rule. A workable procedure is to count threshold crossings over a representative period, and to compare that count with the number of transitions the rule is currently producing. A rule whose firing condition is crossed thousands of times in a week while the underlying service has one problem is a rule that needs a hold duration or a second threshold; a rule whose condition is crossed twice in a week and fires twice needs nothing.

Crossing counts also expose a rarer pathology: a series that alternates between present and absent. This is not an oscillation across a threshold but a change in what the rule can see, and it belongs with the state machine’s treatment of missing data rather than with hysteresis. The alert clears when the series disappears, which is correct only if the disappearance has its own explanation; Alerting on Absent and Stale Data treats the case where absence is itself the signal.

The last constraint is that all of these timers are additive, and the sum is the response latency. Evaluation interval, hold duration, router grouping delay, and escalation timeout all sit between the onset of a condition and a human acting on it. A rule configured with generous values in each of the four places can take several times the length of the incident it was built to catch before anyone sees it, and no single configuration file shows that total.