Reconstructing an Incident From Partial Evidence

An incident ends, the system recovers, and the record that remains is a set of fragments: a metric that stepped, a log line that stopped appearing, a deploy that finished at some minute, a client report that arrived two hours later. The reconstruction is the model that makes those fragments consistent with one another, and every model carries assumptions that are not in the fragments. The difference between a reconstruction that supports action and one that merely sounds coherent is whether those assumptions are stated.

Fragments have fidelity, not just content

Each source of evidence has a resolution and an error, and the two are independent.

Logs are point events with host-local timestamps. Their resolution is limited by the timestamp format, which is microseconds or nanoseconds depending on the logger, but their accuracy is limited by the clock, and their completeness is limited by the level threshold and by whether the process survived long enough to flush. A log line written to a buffer and lost when the process was killed is a fragment that does not exist, and its absence is more likely during a failure than during normal operation.

Traces are per-request structures with durations measured locally and a causal tree carried explicitly. Their completeness is limited by the sampler, and their coverage is limited by which boundaries were instrumented: a trace shows the time inside spans and says nothing about the gaps between them, which is where queueing, network transit, and uninstrumented libraries live. What a retained trace set can and cannot support as a rate claim is a subject in itself, covered under how sampling rules decide which evidence survives.

Metrics are aggregates sampled on an interval. They are the most reliable for establishing that something happened and the least useful for establishing what it was, because the aggregation removed the identity. Their resolution is the scrape interval, and a metric that reports a rate computed by the collector inherits both the scrape interval and the collector’s own failure modes.

Control-plane records are the slowest and often the most decisive. An API audit record of a configuration change carries the identity of the changer, the previous and new values, and a timestamp from a single clock, which makes it unusually trustworthy for ordering. It is also the source most often omitted from the reconstruction, because it lives outside the telemetry system.

Building a timeline means merging these sources, and merging means confronting the fact that their clocks differ. An event that appears to precede another by 200 milliseconds may follow it, if the second source’s clock is ahead by half a second. The order that survives is the causal one: a record written by the callee before returning, a request identifier that appears in both, a control-plane change whose effect must precede the first request that observed it. Where no causal link exists, the ordering is an inference from timestamps and inherits their error.

The gap where the telemetry was

The interval of greatest interest is frequently the interval with the least data. A process that exhausts memory is killed before it can export its final batch. A network partition severs the path to the collector at the same moment it severs the path to the database. A disk fills and writes fail for the logger before they fail for the application’s own work.

The reconstruction has to represent this explicitly rather than treating it as a period of calm. The signature is a cessation: a series that stops rather than dropping to zero, a log stream with no entries for an interval whose end coincides with a restart, an exporter whose own self-metrics show a queue that stopped draining. Distinguishing “stopped” from “zero” is the single most valuable habit in this work, and it is why systems that emit a heartbeat independent of their main work make reconstruction markedly easier.

The second-order consequence is that the reconstruction’s confidence is lowest exactly where the question is sharpest. A timeline that is detailed up to the moment of failure and blank afterwards describes the approach to the failure, not the failure. Treating the blank as the known interval, because everything around it is known, is the characteristic error.

Latent conditions and the trigger

The single-cause narrative compresses an incident into one event, and the compression loses the structure that determines what would prevent a recurrence. Complex systems are described more accurately by a set of latent conditions that make a failure possible, plus a trigger that makes it actual.

A service that handles a null field incorrectly has a latent defect from the day it was written. It becomes an incident when a producer starts emitting nulls. The producer’s change is the trigger and it is not sufficient: without the latent defect, the nulls would be harmless. Fixing only the producer leaves the defect; fixing only the defect leaves the system intolerant of a legal input. The reconstruction that names both is more useful than one that names the later of the two, even though the later one is the one that appears in the timeline.

The same structure appears at other scales. A retry policy without jitter is latent; a dependency’s latency spike is the trigger. A pool with no upper bound is latent; a traffic shift is the trigger. A health check that passes while the process is degraded is latent; a partial failure is the trigger. In each case the timeline shows the trigger and the code review shows the latent condition, and the reconstruction is incomplete until both are in the model.

Cook’s account of complex systems adds the observation that operators are part of the system’s control loop, not outside it. The actions taken during the incident, including the ones that made it worse, are responses to the state as it appeared at the time, and reconstructing the incident without reconstructing what was visible to the responder produces a narrative in which the response looks inexplicable.

Checking a reconstruction against itself

A reconstruction can be tested the same way a hypothesis is tested: derive a consequence and look for it. The consequences that are cheap to check are the ones about things that should be present and are not, and about things that should be absent and are.

If the model says a rollback occurred at 14:32, the version label in request telemetry should change at that time, and processes started before it should stop receiving traffic shortly after. If the model says a connection pool was exhausted, the pool’s own instrumentation should show acquisition waits that account for the observed latency, and the sum of wait time and handling time should approximately equal the client-observed duration. If the model says a dependency timeout propagated, the client-side error should carry the dependency’s error, or the timeout duration should match the configured value.

The check that most often fails is arithmetic consistency. A reconstruction that attributes a 900-millisecond latency to a 120-millisecond database call, with the remaining time unaccounted, is not a reconstruction; it is a partial one, and the unaccounted time is a fragment in its own right. Following the residual is usually more productive than deepening the explanation of the accounted part.

The second check is whether the model predicts the recovery as well as the failure. Incidents recover for reasons, and a model that explains the onset but is silent about the recovery is missing a mechanism. Recovery that coincides with a restart, a traffic decline, or the expiry of a cache entry each implicates a different part of the system, and the recovery is often more informative than the onset because the state that ended it is identifiable.

Stating confidence without collapsing it

The output of the reconstruction is a set of statements at different levels of support, and flattening them into a single narrative destroys the information a reader needs to act. Three levels are usually enough: observed, meaning a record exists that directly shows it; inferred, meaning the model implies it and no observation contradicts it; and unresolved, meaning the evidence is consistent with several mechanisms that cannot be separated.

The mixing is the failure. An unresolved mechanism written in the same declarative voice as an observed event becomes a fact in the next retelling, and the next investigation starts from a premise that was never established. The remedy is mechanical: each statement in the timeline carries its level and its source, and the level is not upgraded because the statement is repeated.

A related discipline is to record the alternatives that were eliminated and how. A reconstruction that says “the connection pool was not the cause, because pool wait times were under one millisecond throughout” is stronger than one that simply omits the pool, and it prevents a later reader from re-running the same elimination. This is the same principle as recording which probe eliminated which hypothesis, applied after the fact.

Where the reconstruction stops

The investigation ends when additional work would not change the action to be taken. That is a different stopping condition from “the model is complete”, and the difference is worth keeping in view, because a model can always be refined. A reconstruction sufficient to justify a specific change to a timeout, a limit, or a retry policy has met its purpose even if the exact interleaving that produced the first failure remains unresolved.

The opposite failure is stopping at the first mechanism that explains the symptom. The symptom is usually the intersection of several conditions, and a model that accounts for one of them will predict a fix that does not hold. The test is whether the model can explain why the failure was rare, why it started when it did, and why it ended when it did; a mechanism that answers one of those questions and not the others is a candidate, not a cause.