Structuring a Game Day: Scenario, Roles, and Safety Nets

An experiment asks whether the system behaves as claimed. A drill asks whether the people and the procedures that operate it behave as claimed when something goes wrong. The two need different designs, and a game day that tries to be both at once usually fails at one of them: the response is disturbed by the injected fault being unknown, or the fault is distorted by the responders intervening in ways that hide the system’s own behaviour.

Deciding which of the two is being run

An experiment’s output is a comparison between a hypothesis and a measurement, as described in formulating the steady-state hypothesis, and it is indifferent to who is watching. A drill’s output is a timeline of human actions and the information available at each decision point. The measurement for a drill is time to detection and time to mitigation, and the interesting part of the record is not how long the whole thing took but where the two numbers diverge: a long detection time is a monitoring gap, a long mitigation after a short detection is a procedure gap, and a short detection with a wrong first response is an information gap.

The design differs because the two tolerate different amounts of interference. In an experiment nobody should touch the system while the fault is active, since an operator action changes the variable under test. In a drill the responders are supposed to act, and the exercise intends to observe whether their actions improve the situation. Running both at once is possible but requires the fault to be announced to the responders; running a blind drill and treating it as an experiment measures neither cleanly.

Blindness deserves a decision rather than a default. Withholding which dependency has been broken tests diagnosis and the honesty of the monitoring, since a team that only knows the fault because the injector told them has learned nothing about their detection. Announcing the fault tests remediation speed and the correctness of documented procedures, and it avoids the failure mode in which an entire exercise is spent on a diagnosis that the fault’s own quirks make easy or impossible. Compound scenarios with two simultaneous faults are the most demanding of all, and they invalidate attribution: when the system breaks, there is no way to know which injected fault caused it, which is acceptable for a drill about coordination and unacceptable for one that is meant to produce findings about a mechanism.

The scenario document

The scenario is written before the exercise and constrains the fault, not the response. It names the failure to be injected, the target scope and how targets are chosen, the magnitude, the intended duration, the abort condition, and the reversion action. Anything that specifies what the responders should do is a procedure, not a scenario, and mixing the two produces a scripted exercise that measures compliance with the script rather than the system.

Three properties of the document do most of the work. The target scope is stated in terms the injector can verify — a specific set of instances selected by the deterministic partition described in bounding the blast radius — rather than as a percentage to be interpreted at run time. The duration has both a planned value and a hard ceiling, so that an inconclusive exercise still ends. The abort condition is expressed in the same signals the responders will be watching, so that the same evidence that would end a real incident can end the exercise.

The environment matters as much as the fault. An exercise run in a staging environment has no users to harm and no fidelity: the traffic, the data volume, and the dependency latencies differ enough that failure paths behave differently, and a fault that a distributed system absorbs under real load may be fatal in a synthetic approximation. An exercise run against live traffic is the reverse trade, and it is only defensible when the fault is small, the abort path has been tested, and whatever error budget the exercise consumes is spent deliberately rather than discovered afterwards.

Roles, and the single authority to stop

Four roles are enough for most exercises, and one person may hold more than one of them in a small setting, but the stop authority must belong to exactly one.

The facilitator owns the scenario and the clock, decides whether the exercise continues, and holds the authority to halt it. The injector applies the scripted fault and nothing else; an injector who improvises a second fault to make things “more interesting” invalidates the record and usually violates the exercise’s agreed scope. Responders behave as they would in an incident, including escalating when they would normally escalate. Observers take a timeline and do not act. A communications role keeps people outside the exercise informed so that they do not escalate it as a real incident, and so that an external escalation is recognised as a signal that the exercise has escaped its boundary.

Two distinct stop concepts need separate names. An operational stop aborts the injection and begins reversion, and it may be invoked by the facilitator or by anyone with evidence that the abort condition has been met. An exercise stop ends the drill entirely, after which the responders are told the exercise is over. Confusing them is a common source of confusion during the drill itself, since a stop that only halts the fault leaves the responders still believing they are in an incident.

Safety nets that must exist before the first minute

The abort path is tested before the fault, as a separate operation with its own verification: apply the fault, revert it, confirm the system is back to its baseline state. An untested reversion is an assumption, and the moment it is needed is the worst moment to discover that the rule, the limit, or the configuration file cannot actually be restored.

The change window is frozen for the duration. A deployment during a game day makes every subsequent observation ambiguous. If the fault consumes error budget, that spend is authorised in advance by whoever owns the budget, because an exercise that silently consumes the month’s remaining tolerance for real failures is a liability regardless of what it teaches. Capacity has a floor as well: the injected fault must not consume redundancy that something else will need while it is active.

The final net is the truth protocol. Everyone who might interact with the affected system needs to know that an exercise is running, and anyone who might page the responders needs to know how to distinguish the exercise’s traffic from real failures. That distinction is a telemetry problem before it is a process problem; without markers that separate injected faults from organic ones, the exercise’s own alerts become a second incident, and the notification path is precisely where the two get conflated.

# the watchdog is the last resort: it runs outside the exercise's control
# plane and reverts the fault unconditionally when the deadline passes
deadline=$(( $(date +%s) + 1800 ))
while [ "$(date +%s)" -lt "$deadline" ]; do sleep 5; done
tc qdisc del dev eth0 root 2>/dev/null || true
test ! -e /sys/fs/cgroup/experiment/memory.max || echo max > /sys/fs/cgroup/experiment/memory.max
echo "watchdog reverted the exercise at $(date -Is)"

The timeline is the artifact

The record kept during the exercise is what survives it, and it is worth being specific about what to capture: the moment the fault was applied, the moment the first symptom was visible in telemetry, the moment a human noticed, the first hypothesis formed, the actions taken and the time each took effect, the moment the abort fired, and the moment the system returned to its baseline. Each of those is a timestamp and an actor, not an opinion, and the intervals between them are the measurements.

The review afterwards examines decisions in the light of the information available at the time rather than in the light of what was learned later, which is the difference between a review that produces changes and one that produces defensiveness. A recommendation is only complete when it lands as something that can be tested: a detector that would have fired earlier, a runbook step that was missing, or an automatic action that removes the decision entirely. Closing the loop means re-running the exercise after the change, because a gap closed on paper is a hypothesis about the next drill, not a result of this one. Exercises that are never repeated describe a system on one afternoon; the drift in configuration, traffic, and personnel between quarters is larger than most teams expect, and it is the reason a game day belongs to a recurring calendar rather than to a project plan.