Bounding Blast Radius: How a Fault Injection Chooses Its Targets

A fault that is injected everywhere is not an experiment, it is an outage with a schedule. The reason to bound the fault is not timidity: containment is what makes the result interpretable. When a subset of targets receives the fault and the rest does not, the untouched subset becomes a control group, and the difference between the two populations isolates the effect of the fault from the effect of everything else that moved during the exercise.

The three axes of containment

Containment has three independent dimensions, and any one of them can be widened far enough to remove the safety margin regardless of the other two.

The first is the set of targets: which hosts, replicas, cells, or tenants receive the fault. The second is the share of traffic or state affected: a fault that hits one replica out of six affects roughly a sixth of capacity, but a fault that hits one replica while the fleet is already running at eighty percent utilisation leaves no headroom anywhere. The third is duration, which interacts with the first two through recovery time. A fault lasting longer than the time to rebalance, restart, or fail over tests the steady-state of the degraded configuration rather than the transition into it, which is a different and often more revealing property.

For any fault the three multiply into a risk estimate that can be compared against a tolerance. Killing two of five replicas of a consensus-replicated store is safe by construction, because a quorum of three of five survives; killing three is not, because the majority is gone and the remaining nodes cannot commit writes. The arithmetic is worth stating explicitly in the experiment design, because it is the same arithmetic that decides whether the fault is even meaningful. Raft-style replication requires a majority for progress, so a cluster of 2f+1 nodes tolerates f simultaneous failures; an experiment that kills f+1 tests the failure path, and an experiment that kills f tests the recovery path. Both are legitimate, and confusing them produces experiments whose results nobody can act on.

Selecting targets deterministically

Random selection is simple and unstable: a different subset each run means successive runs cannot be compared, and a target that is unlucky twice in a row sees the fault twice in a row. Deterministic partitioning gives a stable subset and makes it possible to ask whether the same host is repeatedly implicated.

The usual construction is a hash of a stable identity, taken modulo the number of buckets, with targets assigned to a bucket. Any hash function with good avalanche properties works, including the ones in the standard library.

import hashlib

def bucket(identity: str, buckets: int = 100) -> int:
    digest = hashlib.sha256(identity.encode("utf-8")).digest()
    # first 4 bytes as a big-endian unsigned integer, then mod the bucket count
    return int.from_bytes(digest[:4], "big") % buckets

targets = [h for h in hosts if bucket(h) < 10]   # a stable 10 percent

Two properties matter. First, the identity must be stable across restarts and reschedules; a container name that changes on every deploy breaks the assignment. The instance identifier or the hostname of the underlying node is more durable. Second, the bucket count sets the granularity of the smallest step, so a hundred buckets cannot express a one-percent selection; a fleet smaller than the bucket count cannot be split at all, and the selection degrades to all or nothing. For a three-node cluster the only available splits are zero, one, two, and three nodes, and the “ten percent” of a larger fleet has no analogue.

Shadow or canary traffic models offer an alternative containment in which the fault is applied by request attribute rather than by host: requests carrying a designated tenant, a test user, or a specific header are routed through the faulted path while ordinary traffic is untouched. This works only when the request path is separable, and it fails when the state is shared — a database row locked by a faulted request blocks an untreated one, so containment at the request layer does not imply containment at the resource layer.

The dilution problem in aggregate measurements

Suppose the steady-state hypothesis of the experiment that defines normal behaviour is evaluated over a service-wide error ratio while ten percent of instances are faulted. If the fault pushes the treated instances to a fifty percent failure rate, the aggregate moves by five percentage points — visible, but small enough to hide inside a bound set at ten percent. The experiment then reports success while the affected callers experienced a partial outage.

There are three workable answers. Scope the observable to the treated subset, which requires the telemetry to carry a target label that the injector also knows. Report both scoped and aggregate values and treat the scoped value as the hypothesis. Or use a comparison between the treated and untreated populations, which tolerates workload drift and is the form a request-layer injector naturally produces.

Reporting both values also exposes the second-order effects that containment hides: if the aggregate degrades more than the treated fraction can explain, the untouched targets are being harmed by something other than the direct fault, which is usually a shared dependency or a retry storm.

Ways a bounded fault escapes its bounds

Containment is a property of the system’s topology, not of the experiment’s parameters, so several paths widen the fault beyond the selected targets.

Shared dependencies are the most common. A faulted instance that holds a connection pool on a shared database can consume connection slots until the untouched instances cannot connect. The failure then appears in the control group, which is why a control group that degrades is itself a finding. Resource-level isolation, not target selection, is what bounds this, and it is the reason host-level containment and cgroup-level containment are not equivalent.

Retry amplification is the second. A fault that removes one of six replicas shifts its traffic to the other five; if callers retry aggressively, each failed request can become two or three requests elsewhere. The injected fault is bounded but its load is not, and the observable consequence is concentrated on targets that were never selected. Injecting at the RPC boundary makes this visible directly, because the retry behaviour is part of what the experiment provokes.

Third-party dependencies are the third path, and they cannot be partitioned at all. Killing a specified fraction of application instances may still send every remaining request to the same external API, and the external API cannot be told to serve only the control group. Experiments that depend on a shared external resource require either an interposed stub or an explicit decision to accept that the fault is not contained, and in the latter case the only remaining separation is in the analysis, which is why the marker described in separating injected faults from organic failures has to reach the third party’s call sites as well as the internal ones.

Finally, scheduled work escapes time-based bounding. A fault lasting an hour covers a batch job that runs hourly; the same fault lasting fifty minutes does not. Where the system has periodic jobs, the honest bound is expressed relative to their periods rather than as an absolute duration.

Choosing a starting scope

The defensible default is the smallest fault that can still contradict the hypothesis. If the claim concerns failover between replicas, one replica is enough; if it concerns the behaviour of the quorum under partition, the smallest partition that still permits progress is the one that tests recovery rather than unavailability. Scope should expand only after an exercise at the smaller scope has passed, and expansion should be recorded as a change to the experiment rather than treated as the same test with a larger number.

It is also worth deciding in advance what a failure at the current scope implies. A one-replica fault that breaks the service is a finding; a five-replica fault that breaks it is a restatement of the redundancy budget. Only the first is actionable, which is the practical argument for small scopes beyond safety: the smaller the fault, the more a violation says about the system rather than about the size of the fault. Deriving the scope from the redundancy the design claims to have turns the choice from a guess into a check on a stated assumption, and the assumptions are usually documented somewhere in the capacity plan even when nobody treats them as testable.