Anatomy of a Runbook Step

A runbook that consists of a paragraph beginning “restart the service” is not a procedure. It is a hint. The difference shows up at the moment of use: a procedure tells the responder what state the system should be in before the action, what exactly to run, what observable result proves the action worked, and when to stop and escalate instead. Remove any of those and the responder is improvising under time pressure, which is the condition the runbook existed to prevent.

Steps are units of verified change

The useful granularity is the single change with a single verification. A step has five parts: a precondition that establishes the current state, an action bounded in scope, an expected result stated as an observable, a time bound after which the step is considered failed, and an abort condition that names what to do instead.

The precondition is what distinguishes a runbook from a script. Restarting a process is safe when it is not the only replica serving traffic and hazardous when it is. The precondition records which of those the responder is looking at, as a check rather than an assumption: a command whose output shows the replica count, a dashboard panel, a status field. Checks that require interpretation are weaker than checks with a boolean outcome, but both are better than a sentence that assumes the reader already knows.

The expected result is the part most often reduced to “the command succeeds”. Exit status zero means the command was accepted by the system, not that the system recovered. A restart that returns successfully can still leave the process crash-looping, and a failover that returns successfully can still leave the old primary accepting writes. The verification has to observe the property that motivated the step: the error rate falling, the replica reporting healthy, the queue depth decreasing. That observation takes time, which is why the time bound and the verification are linked; the step is not complete until either the observable appears or the bound expires.

Diagnostic and remediation steps have different rules

Read-only steps can be cheap, numerous, and aggressive. Running a query, fetching a log, or checking a configuration costs little and can be repeated without consequence. Diagnostic steps should be ordered by the ratio of information gained to time spent, and the ones that discriminate between hypotheses should come first. A runbook that begins with six commands that all confirm the service is running has spent the responder’s attention without narrowing anything.

Mutating steps carry a different obligation. Each one needs a stated blast radius, a way to reverse it, and a verification that the reversal worked. The blast radius is not always obvious: a configuration change to a shared load balancer affects every service behind it, while a change to one deployment affects one workload. Writing the radius down forces the author to check it, and it tells the responder when a second person should be involved.

Reversibility is a spectrum rather than a binary. A deployment rollback is usually reversible in the sense that the previous version can be restored, though the data written by the new version may not be. A schema migration may not be reversible at all. Steps that cross an irreversible boundary deserve a separate section, a named approver, and a stronger verification than a step that can be undone by repeating it in the other direction.

Idempotency is what makes re-running safe

An idempotent step converges to the same state whether it is executed once or five times. That property matters because responders re-run steps. They re-run when a command times out, when a terminal disconnects, when a colleague takes over, and when they are unsure whether the first attempt took effect. A step that appends a rule, increments a counter, or creates a new resource on each invocation turns a retry into a second incident.

The usual techniques are declaration rather than mutation: set the desired replica count rather than scale by one, apply a complete configuration rather than append a fragment, ensure a resource exists rather than create it unconditionally. Where the underlying operation is not idempotent, the step needs a guard that checks for the effect before performing the action, and the guard has to be part of the step rather than left to the responder’s memory.

set -euo pipefail

# Precondition: more than one replica is serving, so a restart cannot take the service down.
serving=$(check_replicas --service checkout --state ready)
if [ "$serving" -le 1 ]; then
  echo "refusing to restart: only $serving ready replica" >&2
  exit 1
fi

# Action: restart one replica at a time, waiting for readiness between each.
restart_workload --service checkout --max-unavailable 1

# Verification: the property that motivated the step, not the exit status.
for _ in $(seq 1 30); do
  if check_error_rate --service checkout --window 2m --below 0.01; then
    exit 0
  fi
  sleep 10
done
echo "error rate did not recover within 5 minutes" >&2
exit 1

The verification loop is the part that makes the block a step rather than a command. It has a bound, a success criterion tied to the failure that prompted the action, and a failure exit that tells the next step to take over.

Encoding branches without drawing a flowchart

Runbooks need decisions, and flowcharts are a poor container for them: they are hard to diff, hard to read on a phone, and they rot silently when a box is edited without its arrows. Numbered steps with explicit jumps survive better. “If the queue depth is falling, go to step 9. Otherwise continue to step 6” is readable in a terminal, editable in a text file, and reviewable in a diff.

Two constraints keep branching usable. The first is depth: a procedure with more than two or three decision points is usually trying to cover several different incidents in one document, and should be split. The second is that every branch must terminate in an action, an escalation, or an explicit “no action required, close the incident”. A branch that ends in “investigate further” is the point at which the runbook stops being a procedure, and it should say so rather than pretending otherwise.

Escalation triggers belong in the branch structure, not in a closing paragraph. The condition that means “stop and call someone” is a decision like any other: after fifteen minutes without improvement, if the error rate is above some level, if the action would affect more than one region. Placing it at the step where it becomes relevant means the responder encounters it while it is still actionable.

The automation boundary

Any step that is deterministic, read-only, and cheap can be automated without controversy. The interesting boundary is the mutating step. Automating it removes the responder’s opportunity to notice that the preconditions do not hold, and that opportunity is often the only thing standing between a routine remediation and an outage.

The workable split is to automate the detection and the verification, and to keep the decision to act with a human until the step has been executed enough times to be boring. A step that has run correctly in drills and incidents for long enough that no one reads it any more is a candidate for automation; a step that still requires judgment about scope should stay manual, with the judgment written down. Automation that hides the verification is worse than a manual step, because the responder no longer sees the evidence that the action worked.

Drift is the default state

Runbooks age faster than code because nothing compiles them. A dashboard is renamed, a command loses a flag, a service is split in two, and the document that referenced the old shape remains, still plausible, still linked from the alert. The failure is worse than having no runbook, because the responder spends time following a procedure that cannot work and only later concludes that the procedure is wrong.

The countermeasures are cheap and mostly social. Every runbook carries an owner and a date of last verification. Verification means executing it, at least in a non-production environment or in a drill, not reading it. The alert payload links to the specific section, so a broken link is noticed the first time it is used rather than the tenth. And the incident review treats a stale runbook as a finding in its own right, with the same tracking as any other action item; the loop from review back to procedure is described in postmortem to alert change.

A runbook is also a coordination artifact. When an incident has an explicit commander and a separate person executing steps, the runbook becomes the shared script that keeps the two aligned, which is one of the reasons the role structure in incident roles and status cadence makes procedures easier to follow rather than harder. The payload that delivers the runbook link has its own constraints, covered in what a page must carry.