Severity Tiers, Escalation Timers, and the Cost of a Page

An incident tier is a decision rule, not a description. It exists so that at an inconvenient hour a person who did not cause the problem can decide, within minutes, who to wake, how often to communicate, and whether to stop the deployment route. A tier system that requires discussion to apply has already failed at the only moment it matters.

The common design failure is a single ordinal scale, from one to five, whose levels are defined in prose that mixes causes with effects. “Severe degradation of a critical service” cannot be evaluated in a minute, because the definitions of severe, degradation, and critical are all contestable. Splitting the classification into two observable axes resolves most of the ambiguity and makes the assignment mechanical.

Impact and urgency as independent axes

Impact answers who is affected and how completely. A workable set of classes is small: total loss of a function for all users; partial loss, meaning a fraction of requests fail or a subset of functionality is unavailable; single-tenant or single-region loss; correctness or data-integrity defects, including silent corruption that no user has noticed; and latent exposure, meaning the system is vulnerable or degraded with no present user-visible effect.

Urgency answers how fast the situation is changing and whether a deadline binds. A total outage that is stable ranks differently from a partial degradation that doubles the failure rate every hour, because the second will become the first. A defect that will destroy data older than a retention window has a deadline in wall-clock time, and the deadline is what makes it urgent regardless of present impact.

The two axes combine into a matrix rather than a line. Total impact with a stable state is the highest tier. Partial impact that is accelerating, or that will escalate the moment a nightly batch runs, can be ranked alongside it. Latent exposure ranks below both until something makes it exploitable, at which point the urgency axis moves it. The matrix has a useful property that a single scale lacks: disagreements become specific. Two responders who disagree about a tier usually disagree about one axis, and the disagreement can be settled by a measurement, which is the error rate or the rate of change, rather than by seniority.

Deciding from observables

Every cell of the matrix needs a threshold that can be read from dashboards and logs. Impact is measured by the fraction of requests failing or the fraction of tenants affected, both of which are counts. Urgency is measured by the derivative: the change in failure rate over the last interval, or the time until a deadline expires.

The thresholds belong to the service, not to the organisation, because the same absolute error rate means different things at different traffic volumes. A service handling a hundred requests a minute and one handling a hundred thousand have the same percentage impact at wildly different counts, and an organisation-wide numeric rule for “significant” will be wrong for both.

Correctness defects deserve their own class because the volume metrics do not see them. A service that returns wrong balances for one percent of accounts has a hundred percent failure rate for those accounts and no visible error rate at all, so the impact measure has to include a correctness signal, typically a reconciliation or invariant check rather than an HTTP status count.

From tier to timer

A tier is only operational once it binds timers. Each level should specify four of them: how long before an unacknowledged page escalates to a second person, how long before the incident gets a named commander, how often status updates are published while the incident is open, and how long before an unresolved incident is reviewed by someone outside the responding group.

The acknowledgment timer is the one that most systems get right, because it is easy to implement: page the primary, and if no acknowledgment arrives within a configured interval, page the secondary, then the tertiary. The interval should be short enough that the escalation is still useful for the highest tier, on the order of minutes, and long enough that a responder who is walking to a laptop can answer. Deriving it from the response target rather than from convenience is the discipline: if the target for the highest tier is fifteen minutes from detection to human engagement, an acknowledgment timer of ten minutes leaves both the primary and the secondary a chance.

Update cadence is the timer most often skipped. For an incident with broad impact, a status update at a fixed interval, commonly every thirty to sixty minutes at the highest tier, serves two purposes: it forces the responder to keep an external account of state, and it stops stakeholders from pinging the responder to ask for news, which is a page in all but name.

The attention cost of a page

A page has a cost measured in the responder’s recovery time, not in the seconds it takes to read. An interrupted engineer needs a period to return to the previous context, and the interruption lands at the start of an incident, when the responder is also trying to reconstruct what changed. The cost is paid by every page, whether or not it required action.

That asymmetry, high cost per interrupt and zero value for a no-op page, sets the rule: paging is reserved for conditions that require a human decision within the response target. Everything else is a ticket, a dashboard, or a chat notification. Conditions that self-heal, that need no decision, or that are already visible in the next scheduled review do not warrant a page, and paging them trains the responder to triage by reflex rather than by reading, which is the failure mode that turns a real page into a dismissed one.

Alert volume is a design constraint in its own right. A responder who receives several pages in a night cannot attend to any of them fully, which is the same queueing argument that applies to any single-server system: the arrival rate of interruptions must stay well below the service capacity, or the queue of unhandled pages grows and the effective response time collapses. The cycle time discussion applies directly, with pages as arrivals and the on-call engineer as the server.

Alerting on symptoms, not on hypotheses

An alert condition is a claim that something is wrong. The most useful claims are about the user-visible state of the system, because they remain true regardless of which component is responsible. A threshold on the rate of failed requests, or on the fraction of requests slower than a stated latency, is a symptom. A threshold on the queue depth of one internal component is a hypothesis about the cause, and it is frequently wrong: the queue may be deep for a benign reason while the actual failure is elsewhere.

This is not an argument against cause-based alerts; it is an argument about which ones page. Cause-based signals are useful to the responder once the incident is open, because they narrow the search, and the decomposition of recovery time treats that narrowing as its own phase with its own cost. Before the incident is open, paging on a cause means paging on a model, and models are wrong often enough that the responder learns to distrust the pager.

The most reliable symptom thresholds are derived from a service level objective, because an objective states the level of failure the service is allowed to have, and the distance to that level is a quantity with a definition. Burning the entire allowance in a short window is a symptom that needs a decision now; consuming it at the allowed rate is not. How to compute that distance is the subject of error budgets as a release gate.

Tier drift and how it shows up

Two drifts occur in every scheme. Inflation happens when a tier is used to attract attention, so everything becomes the highest tier and the scale collapses to one value. Deflation happens when a lower tier is used to avoid scrutiny, so a real degradation is filed as minor and the record shows a calm month.

Both are visible in the distribution of assigned tiers over a quarter. A healthy distribution is heavily weighted towards the lowest tier with a thin tail upward. A distribution that is nearly uniform, or one in which the tier correlates with who reported the problem rather than with the measured impact, indicates that the matrix is being applied by negotiation.

The structural defence is to make the criteria mechanical and to separate the assignment from consequences for individuals. A responder who over-classifies out of caution should not be penalised, and one who under-classifies should be corrected by the review rather than by the ranking. Where a tier is later found to be wrong, the useful output of the review is a threshold that would have classified it correctly, which converts a judgment dispute into a monitor change.