Routing Alerts by Required Response: Page, Ticket, and Log Line

An alert is a request for a human to do something. The something has a deadline, and the deadline is what determines the channel. A condition that must be handled before the next business day and a condition that must be handled before the next request both arrive as alerts, and sending them to the same place guarantees that one of the two is handled wrongly. Routing is the part of alerting that decides which, and it is a decision about response requirements rather than about severity.

Three response classes

The classification that holds up in practice is by the deadline and by whether the response can be deferred past sleep. Three classes cover the useful space.

A page interrupts a person who is not working. It is justified when the condition is already affecting users, when delay compounds the damage, and when a specific action exists that the responder can take. The interruption is the expensive part: a page at an inconvenient hour costs more than the same work done at a desk, and the cost is paid whether or not the condition turns out to matter.

A ticket enters a queue and is worked in order. It is justified when the condition will become user-visible if left alone for days, and when the remedy is planned work rather than an immediate action. Certificate expiry, disk capacity trending toward a limit, and a dependency announcing a deprecation all fit. The distinguishing feature is that the ticket can wait for a person to be at work without the situation changing character.

A log line or a dashboard annotation is justified when the condition is informative and needs no action at all. It exists so that a responder who is already investigating something else can see it, and so that a later review can reconstruct what happened. This class is not a failure to alert; it is the correct destination for signals that are useful context and not requests.

The failure mode of the classification is a condition that is written into the wrong class. A ticket for a condition that has a two-hour deadline is a page that arrives too late. A page for a condition with a two-week deadline is an interruption that trains the recipient to ignore interruptions, and that training is the expensive part: the response to a real page degrades because the channel has been used for non-urgent traffic.

Why the channel must carry an action

A page that names a condition but not an action is a notification, not an alert. The recipient’s first task becomes diagnosis, which takes an unbounded amount of time and may require access the recipient does not have at that moment. The page then either gets escalated or gets acknowledged and deferred, and both outcomes spend the interruption without producing a response.

An actionable page has four parts: the affected user-facing capability, the evidence that it is failing, the first diagnostic step, and the owner. The first narrows the search, the second establishes that the condition is real, the third removes the cold-start problem at an inconvenient hour, and the fourth determines whether the page should have gone to this person at all. Runbook links belong to the third part; a link to a general dashboard does not, because a dashboard requires the responder to know what to look for.

The test for actionability is whether the page can be resolved by a documented step. If the only documented step is “investigate”, the condition belongs in the ticket class until an action exists. This is a statement about the maturity of the response, not about the severity of the condition, and it means that the class a signal belongs to can change as the runbook improves.

Ownership, escalation, and the shape of a rotation

Routing requires an owner per rule, and the owner must be a group that can act on the condition. Ownership inferred from the metric’s origin is usually wrong: a rule on the error ratio of a public endpoint belongs to the team that owns the endpoint, even though the metric is exported by a proxy that another team operates. Ownership inferred from the alert’s text is worse, because the text is written by whoever wrote the rule.

Escalation exists because a page can go unanswered. The mechanism is a timer: if the first recipient has not acknowledged within a stated interval, the next recipient is notified. The interval is a policy choice, and the two failure directions are both real. A short interval escalates while the first responder is reading the page, which produces duplicate responses and confuses ownership. A long interval leaves the incident unattended for longer than the condition’s deadline allows. The interval should be shorter than the deadline of the condition being paged on, which is the only principled bound available.

A rotation has a second-order constraint that routing decisions affect: the number of pages per shift determines whether the rotation is sustainable. A rotation that pages a responder several times per night produces a responder who is impaired the next day, and the impairment shows up as slower diagnosis rather than as missed pages, which makes it hard to attribute. Reducing page volume is therefore a routing concern, not only an alerting-threshold concern. The classes above are the main lever: moving a condition from page to ticket removes an interruption without removing the signal.

A third element is the handoff. A rotation whose shifts do not overlap leaves a gap in which a page reaches a responder who is about to go off shift, and the condition is then handed over mid-diagnosis. Overlapping shifts by enough time to transfer open incidents costs less than the duplicated diagnosis it prevents, and the overlap is a routing parameter because it determines when escalation can reach a person who is actually on duty.

Deduplication and the repeat interval

Routing happens after the alert has been evaluated, and the router sees a stream of transitions rather than a set of current conditions. A rule that is true for six hours produces a firing event, then repeats depending on configuration. Without a repeat interval, the single firing event is delivered once and the condition is invisible to anyone who joined later. With a short repeat interval, the same condition is delivered continuously and the channel becomes noise.

The repeat interval is the compromise: the condition is re-announced at a period long enough not to be noise and short enough that a responder arriving mid-incident learns about it. It applies per alert instance, not per rule, so a rule matching twenty instances can produce twenty notifications in one cycle unless the router groups them. Grouping by a label that identifies the affected capability collapses those into one notification while keeping the individual instances addressable, and grouping by a label that is unique per instance defeats the purpose entirely.

Resolution notifications matter as much as firing ones. A notification that says a condition has cleared tells the responder that the automatic remedy worked or that the condition passed, and it closes the loop without requiring a dashboard check. A router that suppresses resolution messages leaves the responder unable to distinguish a fixed incident from an unread one.

The interaction between the alert state machine and the router’s timers is where most of the surprising behaviour lives. A condition that flickers across the threshold produces a sequence of firing and resolved events, and the router’s repeat and group intervals determine whether that sequence arrives as one notification or as a stream. Alert State, Hysteresis, and the Cost of Flapping covers the state machine itself; the router’s behaviour is what makes its output tolerable.

When routing cannot be decided per rule

Some conditions have no single owner. A dependency failure affects every service that calls it, and the page has to reach the team that operates the dependency rather than the teams that observe the symptom. Routing by symptom is correct for detection and wrong for ownership in exactly this case, so the rule needs a stated owner that is not derived from the metric’s source. An inhibition rule tied to the dependency’s own alert removes the dependent teams from the notification path for the duration, which is often a better answer than deciding which of them should be woken first. Symptom-Based Alerting and the Cause Signals It Replaces argues for detecting at the boundary; the routing decision adds the separate question of who is woken.

The remaining structural problem is volume: as the number of rules grows, the classification of each one into a response class becomes its own maintenance task, and rules that were correctly classified when written drift as the system changes. A rule whose page no longer corresponds to an action is the common case, and the only reliable detector is a review of what actually paged, which is why the record of past pages is a routing input and not only an incident archive.