Postmortem Causation: Triggers, Latent Conditions, and Failed Defences

An incident review frequently ends with a sentence of the form “the outage was caused by a configuration change”. The sentence is true in the narrow sense that the change preceded the failure, and it is nearly useless for prevention, because configuration changes happen continuously and almost none of them produce an outage. What distinguishes the one that did is not the change itself but the state of the system that received it.

The distinction is not a matter of taste in wording. It determines which follow-up work is possible, and therefore whether the same class of failure returns. A review that produces a cause produces an action item to prevent that cause. A review that produces the conditions and the failed defences produces action items that reduce a whole family of failures.

Why a single root is the wrong unit of explanation

The phrase “root cause” comes from models of failure developed for systems whose components are arranged in series, where a chain of events can be traced back to a first link. Software systems with redundancy are not arranged that way. They have multiple paths to the same function, multiple barriers between an error and a user, and continuous change in the components.

Cook’s description of complex systems includes the observation that such systems run in a degraded mode most of the time: some components are already failing, some safeguards are already bypassed, and operation continues because the remaining paths are sufficient. An accident is not the arrival of a fault into a healthy system. It is the coincidence of several conditions that were individually tolerated. The corollary is the strongest test available to a reviewer: if the suspected cause occurs routinely without an incident, it is a condition that the system usually absorbs, and the review has not yet found the interesting part.

Reason’s model from the study of human error gives the same conclusion in a different vocabulary. Defences are layered, each with holes, and a failure requires the holes in several layers to line up. Naming one layer as the cause removes it from the analysis and leaves the alignment intact.

Counterfactuals, hindsight, and what a reviewer can observe

A causal claim of the form “X caused Y” is usually tested by asking whether Y would have happened without X. In a physical experiment the test is run; in an incident it cannot be, because the same system state cannot be recreated, and in a stochastic system the outcome would differ anyway.

The practical substitute is a judgment about sufficiency: how many conditions would have had to change for the outcome to differ. That judgment is unstable, because a long list of changes would each have prevented the incident, and the one chosen is typically the change closest to the outcome or the one that a person performed. Human action attracts causal attribution disproportionately, which is the mechanism by which reviews produce conclusions like “the operator typed the wrong command”.

Hindsight supplies the rest of the distortion. Once the outcome is known, the probability of that outcome as judged by a reader is much higher than it was for the responder, and the signals that were ambiguous at the time read as clear warnings afterwards. A review that does not reconstruct the information available at each decision point will judge decisions against information the decision maker did not have. The reconstruction has a concrete form: for each decision, list what was on screen, what alerts had fired, and how long the responder had been awake and in context.

Trigger, latent condition, and failed defence

Three categories carry most of the analytical load, and separating them makes the review’s output actionable.

The trigger is the event immediately preceding the failure: a deploy, a feature flag change, a traffic pattern, a hardware fault, a certificate expiry. Triggers are numerous and mostly harmless. Prevention at the trigger is weak, because the space of triggers is open and the next one will be different.

A latent condition is a property of the system that made the trigger consequential: a client without a timeout, a failover path that had never been exercised, a retry policy that multiplied load on a struggling dependency, a quota that allowed one tenant to consume a shared pool. Latent conditions persist for weeks or months and are present in the system whether or not the incident happens. They are also where the leverage is, because removing one reduces the consequence of every future trigger that interacts with it.

A failed defence is a barrier that should have contained the problem and did not: a staging environment whose data did not resemble production, a canary that carried no traffic, an alert whose threshold sat above the failure level, a runbook step that had not been executed since a system change. The distinction between a latent condition and a failed defence is not always clean, but both share a property the trigger lacks: they can be tested. A defence that cannot be shown to work in a drill should be assumed not to work.

Rebuilding the sequence from telemetry

The narrative in a report is a reconstruction, and its quality depends on the ordering of evidence. Timestamped logs, traces, metrics, deployment records, and configuration history are the primary sources; chat and memory are secondary and should be labelled as such.

Ordering across machines is the first technical obstacle. Wall clocks on different hosts drift, because the synchronization protocol corrects them in steps, and a host’s clock can move backwards. Two events a few milliseconds apart on different hosts cannot be ordered by their recorded timestamps. The reliable ordering comes from causal links: a trace identifier propagated in a request header, a sequence number in a queue, a log line that quotes an identifier from another log line. A report that orders events by clock time alone is asserting precision it does not have.

Resolution sets another limit. A metric scraped once a minute cannot distinguish an event at 10:00:05 from one at 10:00:55, so a spike that appears in one sample was somewhere inside that minute. Logs with second resolution do better, and tracing worse still in aggregate, because sampling discards most requests. The report should state the resolution of each source it relies on, so a reader can see which conclusions are pinned by evidence and which are interpolation.

Retention is the third constraint, and it is a planning problem rather than an investigative one. Logs and traces are retained for bounded periods to control cost, so the evidence for an incident discovered late may already be gone. The decision about what to retain for how long is therefore part of the incident capability, not an afterthought of the logging system.

Writing down what is known and what is inferred

A report’s authority comes from separating observation from interpretation. A line that says “at 10:03 the error rate rose from 0.2 percent to 94 percent” is an observation with a source. A line that says “the deploy at 10:01 saturated the connection pool” is an inference, and it should be marked as one, with the evidence it rests on and any competing explanation that was not excluded.

Two conventions help. The first is to write the timeline first, with sources, before any causal statement, and to keep the two sections separate so that a reader can evaluate the evidence without being led by the explanation. The second is to record the alternatives that were considered and why they were rejected, which prevents a later reader from re-litigating a settled question and prevents the review from looking more certain than it was.

Action items that can be checked

The value of the review is realised entirely in its follow-up, and follow-up items fail in a predictable way: they are too abstract to verify. “Improve monitoring” has no completion test. “Add a timeout of the stated duration to the client that retries against the payment service” does, and its completion can be demonstrated by a test that fails before the change and passes after.

A workable item names four things: the mechanism it changes, the failure mode it prevents, an owner, and an observable condition that marks it done. Grouping items by which category they address, trigger likelihood, latent condition, detection, or mitigation, makes the distribution visible. A set composed entirely of detection improvements leaves the underlying exposure unchanged, and a set composed entirely of trigger suppression will be bypassed by the next trigger.

Items also need an expiry. An item that stays open for a year has either been silently deprioritised or was never actionable, and leaving it open misrepresents the state of the follow-up. Closing it deliberately, with the reason recorded, keeps the list meaningful and keeps the decision in view. The related question of how response phases are timed, and therefore which phase an item should target, is treated in decomposing recovery time.