Steady-State Hypotheses and the Abort Condition in Fault Injection

Fault injection becomes an experiment only when it can fail. Injecting a fault and watching the dashboards produces anecdotes; a claim about the system that the fault can contradict produces evidence. The claim is the unit of design, and everything else in the exercise — the fault type, its magnitude, its duration, its target set — is chosen to put that claim under pressure. The claim is the steady-state hypothesis.

The hypothesis is a bounded statement about output

A usable hypothesis names an observable, a workload, an evaluation interval, and a bound. In the abstract: while the system serves a defined request mix, the ratio of failed requests stays below some threshold; the p99 latency of a named operation stays below some threshold; the depth of a queue stays below some threshold. Each clause matters. Without a named workload, a latency figure is meaningless, because latency and load are not independent. Without an interval, a single sample decides the outcome and the result is noise.

The observable should already exist for other reasons. A signal chosen because it is easy to scrape, rather than because a caller would notice it, turns the hypothesis into a restatement of the fault: “the process was killed” is a description of the injection, while “requests to the service succeed at the same rate while a replica is killed” is a claim about the system. The strongest hypotheses are derived from what consumers observe — error ratios, tail latency, correctness of a returned value — with internal signals used as supporting evidence rather than as the pass criterion.

Windows set the resolution of the measurement

Almost every practical hypothesis is evaluated over a window rather than at an instant, and the window decides which faults the exercise can resolve at all. Take a counter that a fault stops from advancing for a period $T$, evaluated with a rate computed over a window $W$. The measured rate over the window deviates from the true rate in proportion to $T/W$: a fault lasting 20 seconds inside a 5-minute window perturbs the average by roughly 7 percent of the underlying traffic change, which may sit entirely inside the noise of a busy service. Sampling and publication intervals compound this. A scrape interval of 15 seconds means the fault must persist across at least one interval to appear at all, and two samples are the minimum for any rate computation to have a value.

The same arithmetic applies to percentiles, and more harshly. A p99 estimated from a few hundred samples per window is dominated by which few samples land in the tail; a window holding fewer than about a hundred observations cannot support a claim about the 99th percentile at all. Practical exercises therefore pick either a longer window with a coarser percentile, or a shorter window with an aggregate that survives small sample counts, such as a mean or an error ratio summed over counters.

A hypothesis written as a query makes the interval explicit and leaves no room for the probe to widen its own window while the fault is running. The error-ratio form below is ordinary PromQL against a counter named for the requests it counts:

sum(rate(http_requests_total{status=~"5.."}[5m]))
  /
sum(rate(http_requests_total[5m]))
> 0.01

The comparison against a constant is the bound; the selector defines what counts as a failure; the range selector fixes the resolution. A fault shorter than roughly one scrape interval can move this ratio only by chance, and the query will report success for a reason that has nothing to do with resilience.

There is a direct trade between resolution and false alarms. A short window detects short faults and also reacts to ordinary jitter, so its bound must be wider, which reduces sensitivity to real degradation. A long window is stable and blind to brief faults. The honest move is to state the window and treat the experiment as being about faults of at least that duration, rather than to claim general resilience.

The abort condition is a different signal

A hypothesis can be violated without the exercise being unsafe. Abort conditions exist for the gap between “the claim is false” and “stop now”: they are guards on the health of the wider system, not on the claim under test. Typical abort conditions are a saturation signal (a queue depth or an error budget burn), an attribution signal (the fault has escaped its intended target set), and a duration ceiling (the injection must end after a fixed time regardless of the readings).

Merging the two concepts is a common mistake in either direction. If the abort condition is set to the hypothesis bound, every experiment ends the instant its result becomes interesting, and the exercise never observes recovery behaviour, which is often the part worth knowing. If the abort condition is absent, a hypothesis violation continues to be injected, and an experiment silently converts into an incident. The standard structure keeps them separate: the hypothesis is checked during and after the injection and recorded, while the abort condition halts and reverts the injection without regard to the hypothesis.

A useful abort threshold is derived from something the operators already treat as a stopping condition — the rate at which an error budget would be exhausted, or a saturation level at which queues stop draining — because that makes the threshold defensible in advance rather than chosen after the numbers arrive. Thresholds set after the fact to whatever the exercise produced are not abort conditions; they are descriptions.

The reversion path must be exercised before the fault is injected. A kill fault reverts by the orchestrator restarting the workload; a network fault reverts when the traffic-control rule is removed; a configuration fault reverts when the previous file is restored. Each of those has a failure mode of its own, and a reversion that has never been run is not a safety net. An abort that fires and then fails to restore state leaves the system worse than a failed experiment would have.

Where hypotheses quietly lie

Counter resets are the most frequent source of wrong conclusions. Increasing counters zero when a process restarts, which is precisely what a kill fault causes, so a raw increase computed across the restart understates the true activity or produces a negative value. Rate functions that compensate for resets handle this correctly, but the compensation depends on detecting the reset rather than on the fault being absent. A hypothesis that counts events per interval is safer here than one that differences a counter across the whole exercise.

Shared signals cause the opposite error. If the observable is aggregated across a fleet that includes both injected and untouched targets, the fault’s effect is diluted by the untouched fraction, and the hypothesis passes because the blast radius was small rather than because the system absorbed the fault. Careful experiments either scope the observable to the injected subset or report both the scoped and the aggregate value, which also makes the target-selection problem described in bounding the blast radius of a fault visible in the data.

Finally, a hypothesis holds over a workload, so it can only be as meaningful as the workload’s stability. Traffic that changes by a factor of three during the exercise moves every latency and error figure with it, and no bound distinguishes the effect of the fault from the effect of the load. Either the injection is placed in a window where the workload is near-constant, or the observable is normalised against a control group that receives no fault, in which case the hypothesis becomes a comparison between two populations rather than a bound on one. The comparison form is more expensive and considerably more resistant to confounders, and it is what makes injectors at the request layer, described in injecting errors at the RPC boundary, easier to reason about than host-level faults: the control population and the treated population differ in exactly one variable.