Fault Injection as a Continuous Regression Test

A one-off experiment answers a question about a system on one afternoon. A test suite answers a question about every revision, which is the property that makes regression testing worth its cost. The distance between the two is mostly mechanical: the experiment has to acquire a pass predicate a machine can evaluate, a fault that applies itself reliably, and a verdict that means the same thing on two different days.

From experiment to test case

The four parts of a test case are the four parts of an experiment, made explicit. A hypothesis states what must remain true, in terms of an observable, a bound, and an evaluation window, as set out in steady-state hypotheses and abort conditions. A fault describes what is injected, at what magnitude, to which targets, chosen deterministically as described in bounding the blast radius so that two runs test the same thing. An abort condition protects the environment. A verdict is a function of the measurements: pass if the hypothesis held throughout, fail if it was violated, and a third outcome — inconclusive — for runs where the environment, not the system, was responsible.

That third outcome is the part one-off exercises never need and continuous suites cannot survive without. A test harness that can only report pass or fail will report environmental noise as regression, and a suite that produces false regressions is disabled by the second week it exists. The harness must be able to detect the conditions under which a run is uninterpretable and refuse to emit a verdict at all.

Failing for the wrong reason

Five environmental causes account for most apparent resilience regressions.

The first is that the system was not in a known state when the fault was applied. If the service was already degraded — a deployment in progress, a dependency slow, a cache cold — the run measures the effect of the fault on a system that was not healthy, and the fault cannot be blamed for the outcome. A pre-check that evaluates the same hypothesis before injecting, and skips the run if it does not hold, removes this class of false failure. The pre-check costs a delay of the evaluation window before every run, which is why long windows make expensive tests.

The second is that the fault was never delivered. An injection can silently fail to apply: a rule attaches to an interface that has been renamed, a selection query matches no targets, a permission changed, or the injector crashed before doing anything. In all of those cases the system behaves perfectly well and the harness records a pass, which is worse than a failure because it is a false assurance. Every run must therefore verify delivery from the system’s own telemetry — the fault’s signature must appear in the signal that the target emits — before the hypothesis is evaluated. A test that verifies only its own exit status is testing the injector.

The third is insufficient duration. A fault shorter than the evaluation window can move an aggregate by less than the noise, which produces a pass for a reason that has nothing to do with resilience. The relationship between fault duration and window length decides what a suite can detect at all, and it should be stated when the test is written rather than discovered by a run that inexplicably returns green.

The fourth is attribution: something else changed during the run. Suites that run against a shared environment inherit every other deployment that lands there. This is the strongest argument for applying faults to a target set selected by a stable hash and for scoping the observables to that set, so that the analysis compares the treated subset with the untreated remainder rather than the environment with itself.

The fifth is state carried from the previous run. Faults that consume a finite resource, such as the exhaustion experiments described in memory, descriptor, and disk pressure, leave the environment in a different state each time they are reverted, so the second run is not a repetition of the first. Resource-heavy faults should either be made idempotent by explicit cleanup and verification, or kept out of the continuous suite and run as scheduled exercises.

set -euo pipefail

# 1. pre-check: the claim itself must already hold before the fault is applied
if ! hypothesis_holds; then
  echo "environment not in steady state; no verdict"; exit 2
fi

apply_fault
sleep "$WINDOW"                    # at least one full evaluation window

# 2. the fault must be visible in the target's own telemetry
if [ "$(injected_signature_samples)" -eq 0 ]; then
  echo "fault was not delivered; no verdict"; exit 3
fi

if hypothesis_holds; then exit 0; else exit 1; fi

Where the suite runs, and what that costs

Running fault injection as part of a merge pipeline is attractive and rarely works. The fault needs time: a window long enough to resolve the fault, a pre-check, a reversion, and a post-check that confirms the reversion. A test of several minutes per case does not fit in a pipeline that already exceeds a reasonable wait, and the environment a pipeline provides is usually a single instance where the very redundancy a fault tests does not exist.

The workable placements are the two that pay for themselves differently. A scheduled run against a staging environment with synthetic load gives a consistent environment and no risk to users, at the cost of fidelity: the staging environment’s traffic shape, data volume, and dependency latencies differ, and some failure paths only occur under the real load. A recurring run against a small, marked subset of production gives fidelity and costs exposure, which is why its faults must be small, its abort path exercised, and its footprint excluded from the indicators, as described in separating injected faults from organic failures in telemetry.

The intermediate design is to keep the synthetic case in the scheduled suite for speed of feedback and reserve the production run for the smaller number of properties that only production can test. Both should record the same fields, so the results are comparable: the experiment identity, the environment, the target set, the duration, whether delivery was confirmed, the measured value against the bound, and the verdict including the inconclusive outcome.

Tests that can no longer fail

A test suite decays in a specific way. A test passes as long as its precondition is satisfiable; when the precondition becomes unsatisfiable, the test still passes, and it is no longer evidence of anything. A dependency was replaced and the fault now targets an address nothing uses. A target selection matches an empty set because the fleet’s naming changed and delivery was never verified. The service was retired and its experiment remained in the catalogue. Or the fault is applied so briefly that the outcome is constant.

Removing a test that cannot fail is part of maintaining the suite, and it is unpopular because the coverage number falls. The alternative is a suite whose size is unrelated to its strength. Verification of delivery is the structural defence, since a fault that is demonstrated to have reached the system cannot be vacuous; everything else is review discipline, plus a periodic check that the fault’s signature changes the observable at all. A test whose injected fault does not move the measurement by a detectable amount cannot detect a regression either, because the regression would also have to be undetectable.

What green does not claim

The suite measures the system against the faults in its catalogue. It says nothing about the faults that are absent: the failure shapes of a dependency that was not included, the layered faults where a slow dependency and a saturated connection pool arrive together, and the correlated faults that a single injection cannot reproduce. A large catalogue is not evidence of coverage, because coverage is a property of the space of failures rather than of the number of tests, and the space is not enumerable. The useful accounting is structural: which dependency types have at least one fault that fails them, which failure shapes — rejection, slowness, silence, and partial failure — are represented for each, and which are consciously absent. That accounting makes the suite’s silence interpretable, which is the most a testing method of any kind can offer.