Alerting on Absent and Stale Data

An alerting rule is an expression evaluated against a time series database, and the expression’s result is a set of series. “No series” and “a series whose value is zero” are different results, and the difference is the difference between a system that is healthy and a system that is not being measured. Every alerting stack that evaluates expressions inherits this distinction, and most of the operational surprises in alerting on availability come from confusing the two.

Zero, absent, and stale

A counter that has not incremented still reports its current value, so a derived rate of zero means the thing genuinely did nothing. That is a measurement, and an alert on it should behave like any other. A series that is absent means the collection path produced no sample: the target was not scraped, the exporter did not expose the metric, or the metric’s label set changed shape. A series that is stale means the collection system once had the series and has since observed the target’s disappearance.

The distinction matters because expressions propagate absence. A ratio of two rates, where the denominator’s series is absent, evaluates to an empty result, not to a large number and not to zero. An alert rule comparing that result to a threshold therefore produces no pending instance, no firing instance, and no notification. The failure is silent in the direction that matters most: the worse the collection problem, the quieter the alerting.

The usual cause of absence in a pull-based system is a failed scrape. The collection system typically exposes a synthetic per-target series whose value is one when the last scrape succeeded and zero when it failed, which makes the failure representable as a value rather than as a gap. That series is the most valuable one in the system, because every other series depends on it and nothing else reports its loss. Its own absence, which happens when the collection system itself is down, is the case that needs the separate mechanism described below.

The usual cause of staleness is a target that stopped being discovered. The collection system marks the target’s series stale, and queries over a lookback window that crosses the staleness marker return nothing. The default lookback window in Prometheus is five minutes, so a series that stopped being updated less than five minutes ago may still be returned from the last stored sample, and only after that does it disappear. This is why an alert can appear to lag when the exporter dies: for up to the lookback window the rule is evaluating a value that no longer changes, and the value may be well within the threshold.

Making absence an explicit condition

The way to alert on absence is to write an expression whose result is present exactly when the data is missing. Prometheus provides functions for this: absent() returns a single element with value one if the given selector matches no series at the instant of evaluation, while absent_over_time() performs the same test over a range and returns one when the selector matched nothing at any point in that range. The range form is the useful one for detection, because the instant form can be satisfied by a single sample arriving between two evaluations.

# Fires when the checkout exporter has not been scraped for 10 minutes.
absent_over_time(requests_total{job="checkout"}[10m])

The result carries no labels from the missing series, only the labels of the equality matchers in the selector, which is what makes the alert instance’s identity stable and its annotations generic. A rule that needs to say which of several targets is missing must be written per target, because a selector matching several targets returns one element for the whole group rather than one per missing member.

The second form of the same idea is a count of healthy members compared against the number that should exist. Instead of detecting a specific missing series, the rule asserts that at least some number of targets are up, which catches the case where the selector’s shape changed and no series matches the original pattern. This is coarser and less dependent on one target’s series shape, and it is the practical way to guard a whole scrape configuration rather than individual targets.

The dead man’s switch

A rule that fires when data is missing fails in exactly the case it exists for if the evaluator itself is down. The construction that covers this inverts the direction: an alert that fires unconditionally and is routed to a receiver whose absence is itself monitored. If the always-firing alert stops arriving, the receiver’s own monitoring notices that a notification which should have been received was not.

The version of this that requires no external monitoring is to route the always-firing alert to a channel that is not the paging channel, using the same router that carries real pages, and to treat the absence of the periodic notification as an incident. The value is not in the notification, which says nothing actionable, but in the round trip: it exercises rule evaluation, the router, and the delivery path continuously, so any break in the chain becomes visible within one repeat interval. That test also covers the failure nobody instruments, which is a rule whose expression was edited into something that matches nothing.

The switch has one requirement that is easy to get wrong: it must fire unconditionally, without a threshold over data. A version written as vector(1) in the alert expression satisfies this. A version that depends on any metric being present reintroduces the dependency it was built to remove.

False resolution

The most damaging consequence of absence in alerting is not a missed page but a wrong one. A firing alert instance is a state machine element that clears when the rule’s expression stops selecting it. If the expression stops selecting it because the underlying series disappeared, the state machine cannot tell the difference between a recovered condition and a vanished one, and it emits a resolution. The responder who acknowledges and moves on has been told that the service recovered at the moment it stopped reporting.

Two mitigations exist, and they are complementary rather than alternatives. The first is to write the rule so that absence does not clear it. An expression that combines the condition with an assertion about the data’s presence, such as requiring that the health series for the target is one, is unsatisfiable when the target is missing, which still produces a false resolution. The construction that works the other way round is to make the alert’s expression true when either the condition holds or the data is missing, which keeps the alert firing through the gap at the cost of conflating two causes in one alert. Separating them into two alerts with a grouping label and letting the router collapse them is usually the better arrangement, since the responder needs to know which of the two is happening.

The second mitigation is to treat the gap itself as an alerting event, as described above, so that a false resolution is immediately followed by a no-data page. The two notifications arrive close together, and their combination tells the responder what happened. This requires that the no-data page be routed to the same place as the original, or the responder sees only the resolution notification and never learns the data is gone. A no-data condition is not a symptom of the service failing, so the response class it belongs in is a decision in its own right, along the lines set out in Routing Alerts by Required Response; a collection failure is often a ticket rather than a page, except where it coincides with the failure it hides.

Absence in the denominator

A distinct case arises when the data is present but the traffic it describes is not. A ratio of errors to requests over a minute in which no requests occurred has no value: the numerator’s rate is zero and the denominator’s rate is zero, and the ratio is undefined. Depending on the query engine’s treatment of division by zero, the result is either an empty vector or a special value, and neither is a health signal.

A service with no traffic is not necessarily healthy, and it is not necessarily broken. It may be intended, as with an internal service whose consumers are all idle overnight, or it may be the first symptom of a failure upstream that has stopped sending requests. Distinguishing them requires a signal that is independent of the traffic, which is the same requirement as the dead man’s switch, applied at the level of the service rather than of the collection system. A synthetic request generator, a periodic health probe from outside, or a heartbeat request from a known consumer all provide it. Without one, the quietest failure mode in a request-driven system is a service that has stopped receiving requests, and no expression over its own counters can detect it.

Alerting on the absence of a signal is closer to instrumentation work than to threshold selection, and it belongs with the same design decision that says which quantities exist at all. The choice of aggregation level is its nearest cousin, since a rule that aggregates across instances can lose the presence of the members even while the aggregate is present; that interaction is treated in Fleet-Wide Alerting and the Per-Instance Rule.