Injecting Failures at the RPC Boundary: Codes, Timeouts, and Resets

Failures at the request boundary are the most direct way to test a caller’s error handling, because the fault is expressed in the vocabulary the caller already speaks. A process kill tests the platform’s recovery; a malformed response or a slow one tests the client library, the retry policy, the deadline arithmetic, and the fallback path — code that is usually written once, rarely exercised, and wrong in ways that only appear when a dependency genuinely fails.

Four shapes that look alike in a dashboard

The first shape is an explicit rejection: the server answers, and the answer says the request failed. At the HTTP level the status code carries the semantics, and the specification groups codes into classes with distinct meanings. A server error is the dependency’s own fault; a gateway error typically indicates that an intermediate hop could not get a usable answer; a service-unavailable response indicates a temporary condition and may carry a hint about when to try again. Application-level codes such as a conflict or a precondition failure mean something entirely different: the request was understood and rejected on purpose, and no amount of retrying changes that.

The second shape is slowness: the request succeeds but arrives after the caller’s patience has expired. From the caller’s side this is indistinguishable from failure until a timeout fires, and the timeout is what defines the failure. A response that takes longer than the caller’s deadline is a failure even though the server did the work, and the work may have had side effects.

The third shape is connection-level failure: a refused connection, a reset stream, or a closed connection. A reset stream in a multiplexed protocol cancels one request without asserting that the whole connection is broken, whereas a connection-level shutdown tells the caller that every stream on it is gone; the two require different client handling, and clients that treat them identically either abandon healthy in-flight requests or continue using a connection that is about to fail.

The fourth shape is silence, in which no response and no connection event ever arrives. It is the same blackhole introduced in packet-level network fault injection, observed from the application rather than from the interface, and it is the shape that finds clients with no deadline at all.

Retryability is a property of the request, not the error

Whether an error may be retried depends on the method and on the side effects the request can cause. The HTTP specification defines methods such as GET, PUT, and DELETE as idempotent, meaning that repeating the request has the same intended effect as performing it once; POST is not, because the intent may be to append a new resource each time it is sent. Retrying a non-idempotent request after a timeout is unsafe in a way the caller cannot observe: the first attempt may have completed and the response may have been lost, so the retry duplicates the effect.

That leaves two honest options for the non-idempotent case, and injection is the way to find which one a system actually implements. The first is to make the operation idempotent at the application level using a client-generated key so the server can recognise a repeat and return the original outcome. The second is to accept the uncertainty and produce a compensating action. A system that retries an unprotected POST under timeout injection will demonstrate the duplication, and the demonstration is more convincing than reading the client configuration.

Retry also needs to respect what the response says about why it failed. Rejecting an overloaded server with a transient status frequently includes guidance about when the caller should try again, and a client that ignores that and backs off on its own schedule can extend the overload. Errors that indicate a permanent condition must not be retried at all: retrying a validation failure multiplies the load with no chance of a different answer.

Deadlines, and how retries multiply work

Deadline propagation means each layer imposes a strictly shorter budget on its downstream calls than it was itself given, so that the terminal layer stops working while its caller still has time to produce an answer. Without that discipline, retries at every layer compose multiplicatively. Three attempts per layer across three layers means twenty-seven requests reaching the leaf for one request at the edge, which is tolerable when the leaf is healthy and catastrophic when it is exactly what the retries are reacting to.

A retry policy has a worst-case duration, and it is worth computing explicitly: the number of attempts times the per-attempt timeout, plus the sum of the backoff intervals, plus whatever connection establishment and DNS resolution cost. If that total exceeds the deadline the caller was given, the later attempts can never produce an answer the caller will accept, because the caller has already given up. The retries become pure load with no possible benefit — precisely the amplification that a fault injection at the boundary provokes and exposes.

// Requires "context", "fmt", and "time".
// Three attempts, each with its own deadline, and a surrender when the
// caller's own deadline has already gone: the retry loop must not outlive it.
func callWithRetries(ctx context.Context, fn func(context.Context) error) error {
	const attempts = 3
	for i := 0; i < attempts; i++ {
		attemptCtx, cancel := context.WithTimeout(ctx, 500*time.Millisecond)
		err := fn(attemptCtx)
		cancel()
		if err == nil {
			return nil
		}
		if ctx.Err() != nil {
			return ctx.Err() // the caller's budget is exhausted; stop retrying
		}
		// exponential backoff: 50ms, 100ms; jitter should be added in practice
		time.Sleep(time.Duration(50<<i) * time.Millisecond)
	}
	return fmt.Errorf("exhausted %d attempts: %w", attempts, err)
}

Jitter is not a refinement here. If a dependency fails in a way that makes every caller retry at the same instant, the retries themselves become a synchronised load spike arriving exactly when the dependency is least able to serve it. Spreading retries across the backoff interval with a random component turns a synchronised spike into a smooth increase. The cost is a slightly longer expected completion time for the lucky-unlucky distribution of attempts, which is generally the cheaper side of the trade.

What circuit breakers and bulkheads reveal under injection

A circuit breaker watches for failures on a call path and stops issuing calls when a threshold is crossed, then probes with a limited number of requests before restoring normal flow. Injection exercises the parts of that state machine that ordinary failures rarely reach: whether the threshold is measured over a long enough window to distinguish a blip from an outage, whether the probe traffic is small enough not to re-trigger the outage, and whether a path that has been open for a long time recovers correctly when the dependency returns. A breaker whose thresholds were tuned against smooth failure rates can behave quite differently against an injected fault that fails all calls to one dependency while leaving the rest of the system healthy, because per-path failure rates that would never trip a global breaker will trip it here.

Bulkheads — separate concurrency pools per dependency — produce a visible signature under injection as well. With a shared pool, a dependency that stops responding consumes every worker in the pool and the failure spreads to call paths that never touch that dependency. With separated pools, the same fault exhausts the pool for one dependency and leaves the rest alone. Injecting a slow dependency and observing whether unrelated endpoints stay within their latency bound is a direct test of that isolation, and it is a cleaner test than provoking real overload.

Making the injected error look like the real one

The injector should reproduce the status codes and timing that the real dependency produces under the same condition, because callers often branch on the specific code and the specific latency. Substituting a generic server error for a transient one exercises a different branch of the caller’s logic, and experiments that do so have reported resilience for behaviour that the real failure would not trigger. Reproducing a realistic failure also means choosing whether the injected failure is per-request or per-dependency: failing a percentage of calls tests the caller’s handling of partial failure, while failing all calls to one dependency tests failover, and the two sit at different points along the containment axes described in bounding the blast radius of a fault. Both belong in a fault catalogue, and they are different experiments with different expected outcomes. The relationship between the injected error rate and the caller-visible error rate is itself worth recording, because it quantifies how much of the failure the system absorbed through retries, fallbacks, and caches — the number that says whether the injected fault was actually felt.