Inhibition and Dependency Suppression Between Alerts

When a shared dependency fails, the alerts it produces are not one alert. They are the dependency’s own alert plus the alert of every service that depends on it, and the number of notifications is the fan-out of the dependency graph. The responders who can fix the condition are the ones who own the dependency, and they do not need to be told about the dependent services; the responders who own those services cannot fix anything until the dependency is restored. Suppression exists to remove the second group from the notification stream, and it is the alerting feature with the highest ratio of benefit to risk.

The mechanism

Inhibition is expressed as a relation between two sets of alerts, distinguished by matching labels. A source matches a set of firing alerts; a target matches a set of alerts that should be muted while a source is firing; and a constraint on label equality decides which targets a given source covers. When the router observes a firing source alert, every target alert whose equality labels have the same values as the source’s is marked inhibited, which means it is retained in state but not delivered to a receiver.

inhibit_rules:
  # While the datastore outage is firing, do not page for the services
  # that observe it, as long as they are in the same region.
  - source_matchers:
      - alertname = "DatastoreUnavailable"
    target_matchers:
      - severity = "page"
    equal: ['region']

Three properties of this mechanism determine its behaviour.

The matching is by label value, not by graph structure. Nothing in the configuration knows that service A calls service B; the relation is declared by naming alert labels, and the equality constraint is the only thing that ties a specific target to a specific source. This is why the equality label must be a dimension along which the dependency is actually shared: a region, a cluster, or an availability zone. An equality constraint on a label that both alerts happen to carry but that does not express the dependency, such as a team name, will suppress alerts that have nothing to do with the source.

Inhibition is dynamic. It applies while the source is firing and lifts when it stops, so a target that is still firing after the source clears becomes deliverable again. If the target’s own state has meanwhile resolved, nothing is delivered, which is correct; if it is still firing, the responder learns that the dependency recovered and something else is still broken.

Inhibition applies to notification, not to state. An inhibited alert is visible in the router’s own view of the alert set, and it transitions through the states described for firing and resolution normally. A responder examining the current alert set sees the targets alongside the source, which is what allows the dependency to be inferred during an incident rather than after it.

Ordering and non-transitivity

Inhibition is evaluated against the set of alerts that are already firing, and implementations differ in whether an alert can be inhibited by a source that is itself inhibited. The safer assumption is that chains are not guaranteed: a three-level dependency where the top-level alert suppresses the middle tier and the middle tier suppresses the bottom tier should not be relied on to suppress the bottom tier. The configuration that works is stated pairwise, from each source directly to the targets it should mute.

Within one evaluation cycle, the set of firing alerts is not fully settled, so two simultaneous transitions can interact. A target that begins firing in the same cycle as its source may or may not be evaluated as inhibited in that cycle, depending on the order in which the implementation applies the rules. The effect is bounded by one cycle, and the practical consequence is that the first notification of a target can escape inhibition during a simultaneous onset. Where that matters, the inhibition should be paired with grouping by the dependency’s identity, so that even an escaped target notification is collapsed into the same group as the source.

The third ordering subtlety is the equality constraint’s direction. The source’s label values are compared to the target’s; if the source does not carry the label at all, the constraint cannot be satisfied and no inhibition occurs. A source alert whose labels are produced by an expression that drops the region label, perhaps because of an aggregation in the rule, will silently fail to inhibit anything. This is the most common reason an inhibition rule appears to do nothing, and it is invisible in the rule’s own text.

Silences and their difference from inhibition

A silence is a human-created, time-bounded filter: a set of label matchers, a start, an end, and the identity of whoever created it. Matching alerts are muted for the interval regardless of anything else firing. The two mechanisms look similar in effect and differ in every operational respect.

A silence is unconditional and manual. It is the right tool for a known planned event, such as maintenance that will take a dependency offline, or for an alert that is known to be wrong while a fix is being deployed. Its risk is that it is unconditional: it cannot distinguish the expected condition from an unexpected one that happens to have the same labels. A silence matching a whole service for a maintenance window also mutes the unrelated alert on that service that fires during the window, and that alert is then discovered after the window ends.

A second risk is duration. A silence that is created for a window and forgotten is a permanent blind spot for the matched alerts, and nothing in the system reports it. Silences therefore need an owner and a reviewable set: a periodic examination of the currently active silences, with an expiry policy for those whose window has passed, is the only reliable detector, and it fits naturally into the ownership and escalation structure described in Routing Alerts by Required Response. Recording the reason and the associated change in the silence’s fields is what makes such a review possible at all.

Neither mechanism should be used to silence a noisy threshold permanently. That converts a wrong threshold into a hidden one, and the correct fix is at the rule.

What suppression costs

Suppression removes information from the notification stream, and the removed information was previously the reason for some of the pages. Two failure modes follow directly.

The first is a suppressed symptom that is not actually caused by the source. If the inhibition’s equality label is coarser than the dependency it expresses, unrelated failures within the same region are muted for the duration of the source incident. The mitigation is to keep the equality label as narrow as the dependency, which for a datastore shared by one cluster is the cluster label, not the region label.

The second is a source alert that never fires. An inhibition rule is only as good as the alert it depends on, and a source alert with an aggressive hold duration delays suppression by the same duration. During the gap, the dependent alerts are delivered, which is the safe direction of failure. The unsafe direction is a source alert that fires for a condition which does not actually affect the dependents, since then real failures in the dependents are muted by an unrelated event. This is why the source should be the outage of the dependency itself rather than a cause signal that merely correlates with it.

A third cost is the loss of coverage evidence. Suppressed alerts do not appear in the notification record, so an analysis of page volume undercounts the number of conditions that actually occurred. Where that matters, the router’s own state is the data source, not the notification history.

Deciding the dependency edges

The relation that inhibition encodes is a directed graph, and the graph has to come from somewhere. The reliable source is the service’s own wiring: the dependency list that the deployment already declares, the connection pool targets, or the client configuration that names the dependency’s address. Deriving edges from correlated alert history is tempting and produces relations that hold during the observation window and are wrong afterwards.

A practical ordering is to add inhibition only where a dependency outage has already produced a fan-out of pages, and to write the rule with the dependency’s alert as the source and the dependent’s alert as the target. Each rule added covers one edge of the graph, and the graph is then as correct as the observations that motivated it. The complementary direction, suppressing a dependent’s cause-based alerts while its own symptom is firing, is the same mechanism applied to the internal structure of a single service; Symptom-Based Alerting and the Cause Signals It Replaces explains why cause alerts should not be in the paging path to begin with.

The last consideration is that suppression and grouping overlap. Grouping by a shared label already reduces the number of notifications for one dependency failure, because all of the dependent alerts land in the same group. Where grouping is sufficient, inhibition adds risk without adding benefit. Inhibition earns its place when the dependents belong to different owners and therefore different routes, because then grouping cannot collapse them and the notification would otherwise reach a team that can do nothing about it.