Alert Fingerprints, Grouping, and Inhibition

A monitoring system that evaluates a rule every fifteen seconds against a thousand targets does not produce one alert when something breaks. It produces the same alert over and over, or a hundred variants of it, or a hundred consequences of one cause. The layer between rule evaluation and a ringing phone exists to answer three questions: which of these firing signals are the same problem, which of them should arrive together, and which of them are explained by something else already known to be broken.

Identity is a label set, not a message

An alert instance is identified by its complete label set. Two evaluations that produce the same labels describe the same alert; the second updates the state of the first rather than creating a new one. This is what makes a firing alert a persistent object with a start time and a resolution time rather than a stream of events.

Implementations usually compute a fingerprint, a fixed-width hash over the sorted label pairs, and index active alerts by it. The hash width matters only for collision probability; the semantics come from the labels. Labels therefore split into two groups. Identity labels describe the thing that is broken and are stable for as long as it stays broken. Payload labels carry detail that may change while the alert is active, such as the current value or the instance that happened to be sampled. Putting a payload label into an alert’s identity is the single most common cause of duplicate pages, because every change in that value creates a new alert with a new fingerprint, and every new alert is a new notification.

The rule of thumb is that an identity label must be stable across the lifetime of the condition and must partition the problem space at the granularity a human would act on. A rule that fires per process on a host with two hundred worker processes has chosen a granularity that no responder wants; the condition is one degraded service, and the alert should say so once with an aggregate count. Conversely, a rule that aggregates away the region has chosen a granularity too coarse to route, because the responder in the affected region is not the same person as the one elsewhere.

Timestamps, request identifiers, and generated object names are the classic offenders. Container names that include a random suffix are a subtler version: the label is stable while the container lives, but the container is replaced during a rolling restart, so a long-running condition generates a new alert per replacement. The fix is to alert on the workload’s identity and carry the instance identifier in the annotation rather than the label.

Grouping changes the unit of notification

Deduplication reduces many evaluations of one rule to one alert. Grouping reduces many alerts to one notification. The router groups firing alerts by a configured set of labels, and the group, not the individual alert, is what gets delivered.

Three timers shape the delivery. The group wait is how long the router buffers a new group before sending its first notification, which trades a small delay for the chance to include related alerts that fire moments later. The group interval is the minimum spacing between notifications for a group that is already open, so newly arriving alerts join an existing message rather than starting a new one. The repeat interval is how often the group is re-sent while it continues to fire. In Alertmanager these default to thirty seconds, five minutes, and four hours respectively, and the asymmetry is deliberate: the first two are about latency, the third is about not letting an unacknowledged page disappear.

route:
  receiver: primary
  group_by: [alertname, service, region]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers: ['severity="critical"']
      receiver: pager
      repeat_interval: 30m

The choice of group-by labels determines whether the notification is useful. Grouping only by cluster produces a message that says fourteen alerts are firing, which forces the responder to open a console before knowing whether the problem is a database or a certificate. Grouping by alert name, service, and region produces a message whose title already states the scope. The cost of finer grouping is more notifications; the cost of coarser grouping is that one notification hides several unrelated failures behind a count.

Grouping also interacts with escalation. Each group typically becomes one incident, so a grouping decision taken at the router determines how many escalation chains start. A failure that produces three groups will page three times even though the underlying cause is one. Choosing the group-by set is therefore a decision about incident granularity, not just about message formatting.

Inhibition suppresses by relationship

Inhibition is different from deduplication and grouping in kind. Deduplication says these alerts are the same; grouping says these alerts should travel together; inhibition says this alert is explained by that one and should not be shown at all.

An inhibition rule declares a source matcher set, a target matcher set, and a set of labels that must be equal between source and target. When a source alert is firing, any target alert whose equal labels match is suppressed. The canonical example is a node-level alert inhibiting every pod-level alert scheduled on that node: the pods are not individually broken, the node is, and paging for each pod adds nothing.

The equality set is where inhibition rules go wrong. If the equal labels are too few, the rule suppresses unrelated failures; a region-wide network alert that inhibits every alert sharing only the service label will hide a genuine application failure in a different region. If the equal labels are too many, the rule never matches and provides false comfort. The labels in the equality set should be exactly those that establish the causal relationship, and no others.

Cycles are the other failure. Two rules that inhibit each other leave the system in a state where neither alert is delivered, and the resolution depends on evaluation order rather than on anything meaningful. A simple acyclicity check over the inhibition graph, treated the same way as a dependency cycle in a build graph, prevents this.

Inhibition is also the correct tool for the alert storm that follows a shared dependency failure. When a database is down and every service that reads from it reports errors, the service alerts are consequences. A rule that inhibits service-level error alerts when the database alert is firing reduces a hundred pages to one. The rule has to be written deliberately, because unlike grouping it is not a presentation choice: inhibited alerts do not appear anywhere except in the suppression record, so an over-broad rule hides real incidents.

Where the model breaks under load

The most common operational failure is a fingerprint that changes faster than the repeat interval. A rule that includes a duration or a rate in its labels, or that fires per ephemeral object, produces an unbounded number of distinct alerts. Each is deduplicated against nothing, each starts its own group, and each may start its own incident. The visible symptom is a page storm whose individual messages all look reasonable, which is why the cause is often misdiagnosed as a monitoring outage rather than as a label design error.

The second failure is flapping. A value oscillating around a threshold produces a resolution and a new firing on each crossing. If the evaluation interval is short, this can repeat indefinitely, and each firing may re-open an incident. A pending duration before firing absorbs short excursions, and hysteresis with separate high and low thresholds absorbs them more completely. The pending duration costs detection latency for genuine failures, so it should be sized against how long the condition must persist before it matters, not against how long the noise typically lasts.

The third failure is the group that never closes. Alerts resolve, but if one member of the group remains firing, the group stays open and the repeat interval keeps re-sending the whole set, including the resolved members. Receivers that show only the notification text, without resolving each alert’s state, present a stale picture. Notification payloads therefore need per-alert state, not just a group title.

Finally, the suppression record matters as much as the delivery. When an alert is inhibited, the only way to know whether the rule was correct is to inspect what was suppressed and when. A system that discards inhibited alerts, rather than recording them with their source, makes the inhibition configuration unauditable and turns every incident review into an argument about whether an alert existed.

What grouping and inhibition cannot do is decide whether the surviving page deserved to be sent. That question is about actionability, and it is measured rather than configured; the feedback loop from pages to alert changes is the subject of measuring alert fatigue. The routing that follows a delivered page, and the timers that govern it, are described in escalation policies as state machines. Both layers assume the alert identity was stable in the first place.