DevOps · On-Call

Severity Classification as a Routing Input

How a severity level selects the channel, the responders, and the update cadence, and why definitions anchored in observable impact outperform definitions anchored in alarm.

Severity is often treated as a label applied after the fact, a word in a ticket that determines how the incident is remembered. In a working on-call system it is an input to routing. The level chosen at declaration decides which channel carries the notification, which people are expected to respond, how quickly an update is due, who may authorize a risky mitigation, and whether the incident produces a written review. Getting the level wrong is not a bookkeeping error; it changes what happens next.

Severity is a contract about response

Each level should be defined by the obligations it creates, not by an adjective. A useful definition names four things: the notification channel, the responder set, the target time to acknowledge, and the cadence of status updates. If a level does not change any of those, it is decoration and should be removed.

This is why severity taxonomies with five or six levels tend to collapse in practice. Two adjacent levels that route to the same channel and the same people will be used interchangeably, and the distinction will drift until nobody remembers it. Three or four levels are usually enough: one that wakes people immediately, one that is handled during working hours with a bounded response, one that is recorded and scheduled, and optionally one for a declared major incident that adds coordination roles and external communication. The last is a qualifier on the first rather than a separate point on the same scale, but organizations often model it as a level because it changes the process so much.

Impact and urgency are separate axes

The two questions that determine severity are how much is affected and how quickly it needs attention. They are independent. A total outage of a non-critical internal tool has high impact and low urgency. A slow leak that will exhaust a disk in three days has low current impact and high urgency once the trajectory is known. A single-tenant failure that affects a contractual customer may be low in aggregate numbers and high in consequence.

Collapsing both axes into one number loses the information that the responder needs. A two-axis assessment, recorded explicitly, lets a level be derived rather than argued about: high impact plus high urgency is the top level; high impact with a workaround and no growth is a scheduled response; low impact with growth is a watch item. The value of writing both down is that disagreement moves from an argument about the level to an argument about the impact, and impact is the part that evidence can settle.

Impact should be measured in the unit the business already uses: fraction of requests failing, number of tenants unable to complete a transaction, regions unavailable, data confirmed lost or corrupted. Urgency should be measured in trajectory: is the condition stable, worsening, or already recovering. A stable condition with a workaround is a different response from an identical condition that is doubling every ten minutes.

Definitions have to be observable

A severity definition that says “significant customer impact” is not a definition, because two responders will disagree about significance and the disagreement will be resolved by seniority rather than by evidence. The definitions that survive contact with an incident are the ones a person can check against a dashboard.

“More than five percent of requests to a paid endpoint failing for ten minutes” is checkable. “Any confirmed loss of durable data” is checkable, though the confirmation may take time. “A feature unavailable to all users in a region” is checkable. Thresholds should be chosen so that the boundary sits where the response actually changes; a threshold that is never close to the observed values is a threshold that has never been tested.

There is a tension between precision and speed at declaration. At the moment an incident is declared, the impact is often unknown. The resolution is to allow declaration at a provisional level based on the worst plausible reading, with an obligation to reclassify once the first evidence arrives, typically within the first update interval. Declaring low and upgrading later costs response time; declaring high and downgrading costs attention. Which error is cheaper depends on the service, and the choice should be made deliberately rather than by default.

The cost of inflation, and of deflation

When every incident is severe, severity stops routing anything. The channel that wakes people is used for conditions that could have waited, responders learn that a page does not imply urgency, and acknowledgement latency grows for the incidents that matter. The mechanism is the same one that produces alert fatigue, and the remedy is the same: a level is only meaningful if the levels below it are actually used.

Deflation is less visible and more dangerous. A team that avoids declaring high severity because declarations attract scrutiny will handle serious incidents quietly, without the responders, authority, or communication that the level would have brought. The visible symptom is an incident that was handled at the wrong level and discovered afterwards in a review, which is precisely the case where the taxonomy existed to help.

Both failures are symptoms of a taxonomy that is being used to judge people rather than to route work. If a high severity declaration produces blame, declarations will be suppressed; if it produces resources, they will be accurate. The review culture that keeps the signal honest is the same one that makes post-incident analysis useful, discussed in postmortem to alert change.

Changing the level during an incident

Severity is not fixed at declaration. Incidents grow and shrink, and the level should follow the observed impact. An upgrade is usually straightforward: whoever is coordinating can raise the level when the evidence supports it, and the routing expands to include the additional responders and the communication cadence.

A downgrade needs more care, because it removes people and slows the cadence. Two conditions make it safe: the impact has been measured after the mitigation rather than assumed, and the person performing the downgrade states what would cause a re-upgrade. A downgrade that happens because the responders are tired, rather than because the impact fell, is how an incident becomes a silent one.

Every change of level should be recorded with a timestamp, the reason, and who made the call. The record is what makes the timeline reconstructable later, and it is also what reveals a taxonomy that is systematically misused: a service whose incidents are almost always upgraded within minutes has definitions that are calibrated too low at declaration.

Severity is not the same as urgency of the alert

An alert can carry a suggested severity, derived from the rule that produced it, and that suggestion is useful for routing the first notification. It is not authoritative. The same metric can represent a trivial condition during a maintenance window and a serious one during a launch, and the alert rule has no way to know which. The responder, or the person coordinating, confirms or overrides the level once there is evidence.

This separation matters because it prevents the alert configuration from becoming the de facto severity policy. Alert rules are edited by the team that owns the metric; severity definitions are a property of the service and its commitments. When the two are the same field, a rule change silently changes the response obligations, usually without anyone noticing until an incident is routed at the wrong level.

The channel and responder set that a level selects are the inputs to escalation policies as state machines, and the cadence it selects is one of the responsibilities of the coordinating role described in incident roles and status cadence. Severity is the join between the two: it is the point where a technical observation becomes an organizational commitment.

References

  1. Site Reliability Engineering, Chapter 14: Managing Incidents — Google
  2. NIST SP 800-61 Rev. 2: Computer Security Incident Handling Guide — NIST
  3. The Site Reliability Workbook, Chapter 9: Incident Response — Google