Measuring Alert Fatigue Instead of Describing It

Fatigue is usually discussed as a feeling: the rotation is heavy, the pager is noisy, people are tired. Feelings are real but they do not compare across teams or survive a planning meeting. The alternative is to treat page load as a measurable property of a service, in the same way that error rate or latency is, and to use the measurement to decide which alerts should stop existing.

Fatigue is a rate, and rates have denominators

The first useful quantity is pages per person per week, or pages per twelve-hour shift. It has a denominator, it is comparable between rotations of different sizes, and it can be tracked over time. Google’s SRE book offers two reference points that make the scale concrete: a maximum of about two events per twelve-hour on-call shift, and a limit of roughly a quarter of an engineer’s time spent on call. Both are guidelines rather than laws, but they establish that the intended rate is low, on the order of a few interruptions per shift rather than dozens.

The second quantity is the distribution across the day. A rotation that receives ten pages a week, all between 09:00 and 18:00, is a different experience from one that receives the same ten at 03:00. Splitting the count into in-hours and out-of-hours, using the responder’s local time rather than UTC, separates the load that can be absorbed from the load that damages sleep. The out-of-hours share is also the number that most directly predicts attrition, and it is the one most often omitted from summaries because it is inconvenient.

The third is concentration. Page counts follow a heavy-tailed distribution: a small number of rules produce most of the volume. Reporting the top ten rules by page count, with their share of the total, is more actionable than any aggregate, because it identifies the specific configurations that would change the rate if they were fixed or deleted.

Actionability is the numerator that matters

Volume alone cannot distinguish a noisy rotation from a busy one. The distinguishing quantity is how many of those pages required a human to do something. An actionable page is one where the responder took an action that changed the state of the system, or made a decision that a human needed to make. A page that was acknowledged, investigated, and found to reflect a condition that had already recovered, or that would have recovered without intervention, was not actionable.

The usual way to collect this is a one-touch classification at the end of each page: action taken, no action needed, duplicate of another page, known and accepted condition. The classification is subjective, which is acceptable because the aggregate is what matters and the categories are coarse. A sample is enough: classifying every page for a month on one rotation produces a usable actionability rate, and if that is too much, classifying a random subset of pages gives an estimate with a stated uncertainty.

The rate is then actionable divided by total, computed per rule and per service. A rule with a high volume and a low rate is a candidate for deletion or for conversion into a dashboard signal. A rule with low volume and a high rate is doing its job and should be left alone, even if its title is ugly. The combination is what directs attention, and neither number alone does.

The measurement traps

Any metric that becomes a target stops measuring what it was introduced to measure. Three traps are common enough to be worth naming.

Time to acknowledge is easy to instrument, because the paging system already records it. As a target, it rewards acknowledging quickly and investigating later, which is the snooze behaviour that breaks escalation. It is a useful diagnostic when it is high, because a high value indicates that the notification is not reaching anyone, but it is a poor target, because the cheapest way to improve it is to make acknowledgement meaningless.

Page count as a target rewards batching and delay. A team that is told to reduce pages per week can reduce them by increasing the grouping wait, which lowers the count while increasing the time before anyone knows about a failure. The count improves and the service gets worse. The metric is only safe when paired with a latency measure for the conditions that matter.

Resolution time as a target rewards premature closure. An incident closed while the condition persists is worse than an incident left open, because the reopening starts the clock again with less context. If time to resolve is tracked, it should be paired with a reopening rate, or with a check that the condition was actually observed to have cleared.

The general principle is that the measurement should describe the work, not instruct the worker. A monthly review that reads the numbers and asks why they look the way they do is more useful than a dashboard that turns them into goals.

What the review meeting does with the numbers

The output of measurement is a list of decisions, not a report. For each rule above a volume threshold, the owning team records one of four outcomes: delete it, downgrade it to a non-paging channel, tune its threshold or pending duration, or automate the response so that no human is involved. Recording the decision matters as much as making it, because the same rules reappear in the next review otherwise.

Deletion deserves to be an explicit outcome rather than a last resort. Many alerts exist because a condition was once interesting, and the condition has since been fixed, made impossible by a design change, or covered by a better alert. An alert that has not produced an actionable page in a defined period is a candidate for removal, and the burden of proof should sit with keeping it rather than with deleting it. The period has to be long enough to cover a seasonal cycle and short enough that the review stays relevant; a quarter is a common compromise.

Tuning is the outcome that requires the most care, because a threshold change can silence the condition entirely. A rule should be changed with a stated expectation of what the new volume will be, and the next review should check whether the expectation held. A tuning change made without an expectation is indistinguishable from a deletion in disguise.

Automation is the outcome that removes the page without removing the signal. A condition that has a deterministic, safe response can be handled by a controller, and the human is paged only if the automated response fails or the condition persists. This is the strongest outcome when it applies, because it reduces volume without reducing coverage, but it requires the same verification discipline as any other remediation, which is the subject of the anatomy of a runbook step.

Why the numbers have to be public

A rotation’s page load is usually invisible outside the rotation. The team absorbs the interruptions, the work is not counted against anything, and the cost appears only as slower feature delivery or as departures. Publishing the rate per service, alongside its error budget and its deploy frequency, moves the cost into the same conversation as the benefit. A service that pages its owners ten times a week is making a demand on the organization that is not visible in a feature plan.

The comparison also creates the pressure that makes deletion possible. A team is unlikely to remove its own alerts without a reason, because each one was added in response to something. A periodic review that shows the rule’s volume, its actionability rate, and its share of the total gives the removal decision a factual basis and a peer context. How the resulting changes feed back into the postmortem process, and how a detection gap becomes a specific alert change, is covered in postmortem to alert change.

None of these measurements is precise. The actionability classification is a judgment, the sampling is imperfect, and the thresholds for review are conventions. The alternative is not a precise measurement; it is an argument about whether the pager is too noisy, conducted from memory, in which the loudest recent incident wins. A rough rate, consistently collected, is a better basis for deleting an alert than any amount of recollection.