From Postmortem to Alert Change: Closing the Detection Loop

A postmortem that produces a list of resolutions without changing an alert, a runbook, or a threshold has produced a document. The useful output is narrower and more mechanical: a timeline precise enough to locate the gap between when the system began to fail and when a human knew, followed by a change to the detection path that can be checked afterwards. Everything else, including the prose about culture, is in service of that.

Two intervals, measured from evidence

The timeline should be reconstructed from machine records wherever possible: metric timestamps, deploy records, alert firing and resolution times, acknowledgement times, and the first message in the incident channel. Memory fills the gaps afterwards and is unreliable about ordering, which is why the scribe’s decision log from incident roles and status cadence matters more than recollection.

From the timeline, two intervals fall out. The first is from the start of user impact to the first human who understood that something was wrong. The second is from that moment to the point at which impact ended or was bounded. The sum is the time the service was degraded, and each interval points at a different part of the system: the first at monitoring and routing, the second at diagnosis and mitigation.

The start of impact is often earlier than the first error log, because a partial failure can degrade a service before it produces errors. The best available estimate is usually a change in a user-facing metric, such as a latency percentile or a success rate, and the timeline should state which signal was used and why it is believed to mark the start. Precision to the second is less important than consistency: incidents reviewed with the same definition of impact onset can be compared, and incidents reviewed with different definitions cannot.

Reporting a single incident’s detection time as a number is misleading, because one incident is not a sample. The useful view is the distribution across incidents over a quarter, with the outliers examined individually. A median detection time of a few minutes with one incident that went unnoticed for an hour is a different situation from a uniformly slow detection path, and the fixes differ.

Detection gaps have four shapes

The first shape is that no signal existed. The failure produced no metric, log, or probe that anyone was watching, and the first indication was a user report. The fix is a new signal, and the design question is what to measure: the user-visible symptom is more durable than the internal cause, because the cause changes when the architecture changes.

The second shape is that a signal existed but no alert was defined on it. This is common with new services, and with metrics that were added for debugging and never promoted to alerting. The fix is an alert rule, with the usual questions about threshold and pending duration, and with the same actionability test as any other page.

The third shape is that an alert fired, but late. The delay can come from the evaluation interval, from a pending duration that was sized for noise rather than for the condition, from an aggregation window that dilutes a short spike, or from a grouping wait that held the notification while related alerts arrived. The fix is a change to one of those parameters, and the timeline should identify which one by comparing the alert’s fire time against the impact onset and against the rule’s own window.

The fourth shape is that an alert fired on time and was not acted on. This is the shape most often misdiagnosed as a monitoring problem. The alert may have been suppressed by an inhibition rule, routed to a channel nobody monitors, grouped into a notification that buried it, or acknowledged and then forgotten. Each of those has a different fix, and none of them is “add another alert”. The measurement practices in measuring alert fatigue exist partly to make this shape visible before it is discovered in a review.

Classify the action before assigning it

Action items improve when they are classified by what they change. Four classes cover most of what a review produces.

Prevention removes the failure mode: a configuration that cannot be set to the dangerous value, a dependency with a fallback, a migration that is reversible. Detection adds or repairs the path from failure to human awareness. Mitigation shortens the impact once failure occurs: a circuit breaker, a failover that is actually exercised, a runbook step that restores service without diagnosis. Repair shortens the recovery after mitigation, usually by making the restoration procedure faster or more reliable.

The classification matters because the classes are not interchangeable and because a review that produces only prevention items leaves the detection path unchanged. A team that fixes the specific bug but keeps a monitoring gap will be blind to the next bug in the same place. Conversely, a review that produces only detection items accepts that the failure will recur, which is sometimes the right call and should be stated as such.

Each item needs four properties to be trackable: an owner who is a person rather than a team, a date, a verification that demonstrates completion, and a link to the failure mode or the gap it addresses. “Improve monitoring for the checkout service” has none of them. “Add an alert on the checkout success rate below the SLO threshold, owner named, verified by replaying the incident’s metric window against the rule” has all four.

An alert change is verifiable or it is a hope

The verification for a new or modified alert is a test that fails before the change and passes after it. Three forms are practical. A unit test can evaluate the rule expression against recorded time series that include the incident window and a quiet window, asserting that the rule fires in the first and not in the second. A synthetic condition can inject the metric values or the failing probe in a non-production environment and confirm that the alert fires and routes to the expected target. A drill can exercise the whole path, from condition to page to acknowledgement, which is the only form that tests the routing and the escalation along with the rule.

The change should also come with a removal criterion for whatever it replaces. Adding an alert without removing the one it supersedes increases volume, and the older alert usually has a name that is easier to remember, so it survives. Stating “this replaces rule X, which will be deleted after two weeks of overlap” makes the deletion part of the change rather than an afterthought.

The same discipline applies to the other classes. A runbook change is verified by executing the runbook, not by editing it; the step-level requirements are described in the anatomy of a runbook step. An escalation change is verified by a test page that traverses the full chain, including the step that previously failed.

Why action items decay

The completion rate of action items falls with time, for reasons that are structural rather than cultural. The people who understood the item move to other work, the context that made it urgent fades, and the item competes with feature work that has a date attached. Items that survive the first month tend to survive indefinitely, appearing in review after review with the same owner and a later date.

Two practices resist the decay. The first is to cap the number of items per incident, keeping only those that address a gap observed in the timeline. A short list with owners and dates is more likely to be completed than a long one, and the cap forces the review to choose the items that matter rather than listing everything that could be improved.

The second is to review the open items on a fixed cadence and to make the decision explicit when an item will not be done. An item that is deliberately not being pursued is an accepted risk, and recording it as such is more honest than carrying it forward as an intention. The risk register, not the postmortem document, is where it belongs.

Blamelessness is a method for finding the gap

The reason to avoid attributing an incident to an individual’s error is not politeness. It is that “human error” is where the analysis stops, and the detection gap is usually one layer below it. An operator who ran the wrong command was working in a system that made the wrong command easy to run, that gave no confirmation of what it would do, and that produced no alert when the result was wrong. Each of those is a changeable condition, and each is a candidate action item in the prevention or detection class.

Richard Cook’s observations about complex systems make the same point from the other direction: failure is intrinsic to systems with many interacting parts, and the defenses that matter are the ones that detect and contain failure rather than the ones that assume it will not occur. A postmortem that ends at the individual has skipped the part of the system that failed to detect, which is the part this entire chain of mechanisms exists to improve.

The loop closes when the next incident’s timeline shows a shorter first interval for the same class of failure. That is the only evidence that the detection change worked, and it is why the review should compare the current incident against the previous one of its kind rather than only against an absolute target. A single incident is an anecdote; a sequence of them, measured the same way, is a trend.