Decomposing Recovery Time into Detection, Diagnosis, and Mitigation

Recovery time is usually reported as a single mean, and the mean is then used to justify a single investment. The number conceals the fact that recovery is a sequence of intervals bounded by different mechanisms, each with its own floor, and that the intervals respond to different changes. Shortening detection by a minute and shortening diagnosis by a minute are not equivalent actions, and a metric that adds them cannot distinguish the two.

The decomposition below is descriptive rather than prescriptive: it names intervals that can be timestamped from existing records, so that the time can be attributed rather than estimated.

Five intervals, each with its own floor

Detection runs from the moment the system begins to misbehave to the moment a person or a checking system is informed. Its lower bound is set by monitoring: a metric scraped every fifteen seconds and evaluated every minute cannot report a fault faster than roughly that resolution, and a condition that requires the threshold to hold for five minutes adds five minutes before the alert fires at all. Notification adds its own delay, from seconds for a push notification to minutes for a queue that is polled.

Acknowledgment runs from notification to the first human action. Its floor is biological and social: a responder who is asleep, driving, or in another incident cannot acknowledge instantly. Escalation timers exist to bound this interval rather than to shorten it.

Diagnosis runs from first action to the identification of a mitigation worth attempting. It is the interval most sensitive to information: the number of services involved, whether recent changes are visible in one place, whether traces can be followed across boundaries, and whether the hypothesis space is small. It is also the interval where a wrong hypothesis costs the most, because a failed attempt returns the responder to the start with less time remaining.

Mitigation runs from the decision to act to the point where the impact stops growing. Its floor is set by the mechanism used: a feature flag takes effect on the next request, a rollback takes as long as the deployment process needs plus the time for old instances to drain, a DNS change takes effect over the record’s time to live, and a failover takes as long as the promotion and the client reconnection require.

Verification runs from the action to confidence that the fault is gone. It has the same floor as detection and for the same reason: the system cannot confirm a recovery faster than it can observe one, so a five-minute alerting window implies up to five minutes of uncertainty after the fix. Reporting recovery before verification produces incidents that reopen.

Why a mean over incidents hides the tail

Incident durations are right-skewed. Most resolve in minutes, a few take hours, and the mean is dominated by the few. Since the mean is computed over incidents rather than over time, it also weights a brief total outage and a long partial degradation equally, although the first may cost far more in user-minutes.

Two statistics are more informative. Percentiles of duration, reported for each phase separately, show whether the tail is a detection problem or a mitigation problem: a long tail in detection with a tight diagnosis distribution points at coverage, while the reverse points at tooling. And impact-weighted totals, computed as affected users multiplied by duration, weight each incident by what it actually cost, which changes the ranking of causes substantially.

The mean also moves for reasons unrelated to capability. A month with two long incidents and a quarter with none will show a large swing with no change in practice, which makes before-and-after comparisons of an intervention unreliable unless the counts are small enough to be inspected individually.

When a second incident arrives

The on-call responder is a single server. A second high-severity incident arriving while the first is open does not get a parallel responder; it gets a queue position. Its recovery time therefore includes the remaining service time of the first incident, and the damage accumulates unmitigated for that entire period.

This is the same arithmetic as any single-server queue, and it produces the same nonlinearity: as the fraction of time the responder is occupied rises, the wait before a new incident is picked up rises faster than linearly. The practical consequences are specific. The mean recovery time reported over a period includes queueing delay that is not a property of the responder’s skill, so it cannot be improved by training. And the largest incidents, the ones that matter most, are often the ones that arrive during another incident, because a shared cause produces both.

Handoff compounds it. A transfer of context at a shift boundary costs time proportional to how much of the state lives in the responder’s head rather than in the incident record, which is the argument for maintaining a written timeline during the incident rather than reconstructing it afterwards.

Mitigation first, understanding second

The ordering of effort during an incident is a design decision that gets made in advance, whether or not anyone states it. Diagnosis-first costs impact, because the fault keeps operating while hypotheses are tested. Mitigation-first accepts an incomplete understanding in exchange for stopping the damage, and defers the causal work to the review described in postmortem causation.

Mitigation-first only works if mitigation paths exist and are reversible. The fastest path is usually to revert the most recent change, which requires that the previous artifact is still deployable, that the data written by the new version is readable by the old one, and that the revert does not itself require a migration. Those properties are not free; they are consequences of how changes are built. Expand-and-contract schema changes, feature flags that separate deployment from activation, and backward-compatible interfaces all exist to keep the revert path open, at the cost of carrying both versions for a period.

Authority is the other half. A responder who can see the fault but cannot authorize a rollback spends the difference between mitigation time and escalation time finding someone who can. Publishing the authority in advance, by change class, converts that interval into a decision made in calm conditions.

Measuring the phases without breaking the comparison

Phase measurement requires timestamps, and the timestamps come from different systems that do not agree on time. Fault onset is usually inferred from the first failing sample, so it carries the resolution of the metric; acknowledgment is recorded by the paging system; diagnosis and mitigation boundaries are recorded by people, which makes them the least reliable and the most consequential.

A practical record stores, per incident: first bad sample, alert fired, acknowledged, first mitigation attempted, impact stopped, verified resolved. Subtracting adjacent fields gives the phases, and the derived values can be aggregated without further interpretation.

from datetime import datetime

phases = [
    ("2025-03-04T02:11:00", "2025-03-04T02:19:00", "2025-03-04T02:21:30",
     "2025-03-04T02:47:00", "2025-03-04T02:52:00", "2025-03-04T02:58:00"),
]

def minutes(a, b):
    fmt = "%Y-%m-%dT%H:%M:%S"
    return (datetime.strptime(b, fmt) - datetime.strptime(a, fmt)).total_seconds() / 60

for onset, fired, acked, mitigate, stopped, verified in phases:
    print(dict(
        detect=minutes(onset, fired),
        acknowledge=minutes(fired, acked),
        diagnose=minutes(acked, mitigate),
        mitigate=minutes(mitigate, stopped),
        verify=minutes(stopped, verified),
    ))

Two cautions apply to any comparison across periods. Improving detection changes the population of incidents as well as their durations, because faults that previously went unnoticed now generate records and because faults that previously appeared late now appear early; a drop in mean duration after a detection change may be a change in what is being measured. And phase boundaries recorded by people drift towards round numbers and towards the narrative the responder expects, so the record is more reliable when the timestamps come from systems and only the decision points come from people.

The interval most often missing from the record is the one before detection, which is exactly the interval that a monitoring investment targets. Where the fault is discovered by a user rather than by the system, the detection interval is unknown and the record should say so rather than substitute the time of the support ticket.