Metastable Failure and the Load That Keeps a System Down

A brief traffic spike saturates a service. The spike ends. The service does not recover, and it keeps not recovering for twenty minutes, with the same request rate that it handled comfortably an hour earlier. Restarting it fixes the problem in seconds, which means the capacity was there the whole time and something was holding the system in a state where it could not use that capacity.

That is a metastable failure: a state that the system can occupy indefinitely, that is not the intended operating state, and that is entered by a transient perturbation but not exited by the removal of that perturbation. The distinction from a normal overload matters because the remedies are different. An overload is fixed by reducing load. A metastable failure is fixed by changing the system’s state, and reducing load may not be sufficient.

Two stable states at the same load

The clearest model is a system with two equilibria. In the healthy one, most requests succeed quickly and the system’s internal work is proportional to the useful work. In the failed one, most requests fail or time out, and the system spends its capacity on work that produces nothing.

A queueing system with retries has exactly this structure. Consider a service with capacity C and offered load λ. In the healthy state, the retry rate is near zero and the effective load is λ, which is below C. If a perturbation pushes the failure rate up, retries add load, which raises latency, which raises the failure rate, which raises retries. The loop has a fixed point at high failure rate and high load, and that fixed point is stable. Removing the original perturbation does not move the system out of it, because the retry load alone now exceeds what the system can serve at the failure rate it is experiencing.

The condition for bistability is that the retry-amplified load at the failed equilibrium exceeds capacity while the load at the healthy equilibrium does not. A system with no retries has one equilibrium and cannot be metastable in this way. A system with bounded retries and no retry budget has two, and the boundary between them is crossed by the transient.

The mechanisms that create the second state

Retries are the most common and the most studied. The amplification arithmetic is in Retry Budgets and the Load Multiplication They Prevent; the metastable consequence is that amplification is not just a multiplier applied to a fixed base, it is a feedback loop whose gain determines whether a second equilibrium exists at all.

Cold caches are the second. A cache that is warm absorbs most reads and leaves a small fraction for the backing store. If the cache is emptied, by a restart, an eviction storm, or a failover to a replica with no cache, the full read rate reaches the backing store, which is sized for the miss rate. The backing store saturates, its latency rises, requests time out, and the timeouts mean the cache entries that would have been populated are never populated, because the request that would have populated them failed. The cache cannot warm while the system is failing, and the system cannot stop failing until the cache warms. This is a genuine deadlock between the two states, and it is why cache warming on restart exists as a mechanism.

Connection pools and their exhaustion are the third. A pool with N connections serving requests that each hold a connection for T supports N/T requests per second. If T rises because the backend is slow, throughput falls proportionally, and the queue in front of the pool grows. Requests waiting for a pool connection hold whatever else they hold: a thread, a request slot, a memory buffer. The pool’s own queue is bounded, but the resources the waiters occupy are not, so a slow backend converts into a caller that is holding capacity for a dependency it cannot reach.

The fourth is the one that is easiest to overlook: the work of failing. Generating an error response, writing a log line, emitting a metric, and incrementing a counter all cost something. A service that normally produces a thousand responses per second and one error per minute can produce a hundred thousand error responses per second under failure, and each one is more expensive than the success path it replaced. Logging at that rate can saturate the log collector, which blocks the handler, which raises latency further. The capacity available for useful work shrinks exactly when the failure rate rises, which steepens the feedback loop.

Why the system does not fall back

A metastable failure persists because nothing in the system prefers the healthy state. Each component makes a locally reasonable decision given its current input, and the sum of those decisions maintains the bad equilibrium.

A client seeing failures retries, which is locally correct. A server seeing a full queue accepts and queues rather than rejecting, because rejecting is a visible failure and queueing looks like it might succeed. A load balancer seeing slow responses keeps sending traffic to the slow instance, because the health check passes: the instance answers, just slowly. A breaker that would have opened is tuned with a threshold that the sustained failure rate hovers just below.

The pattern in each case is a decision rule that treats the current state as the input and does not consider the effect of the decision on the state. Retrying is correct for an isolated failure and catastrophic for a correlated one. Queueing is correct for a burst and catastrophic for a sustained overload. The decisions are individually defensible and collectively stable in the wrong place.

The 2021 HotOS paper by Bronson and colleagues names this class and catalogues the mechanisms, and the important structural observation is that metastable failures arise from feedback loops with insufficient damping. The damping is what the reliability mechanisms provide: retry budgets cap the loop gain, admission limits cap the queue, and breakers remove the load from a failing component.

Breaking the loop

Three families of intervention, in increasing cost.

Reduce the loop gain. A retry budget that caps retries at a fraction of successes removes the high-failure-rate equilibrium entirely, because the amplified load can no longer exceed capacity. This is a structural fix and it prevents the failure rather than recovering from it. It requires that every client participate, since one client without a budget can supply the load that the others’ budgets were preventing.

Remove the load. Dropping traffic below capacity lets the system drain, but only if the dropped traffic is not immediately replaced by retries. Shedding at the edge, with Retry-After, and with retry budgets that prevent the shed requests from returning, is the combination that works. Shedding alone against an unbudgeted client population does nothing, because the shed requests return as new arrivals.

Change the state directly. A restart clears in-memory queues, abandons in-flight work, and resets any accumulated state. It is the intervention of last resort because it discards the work in progress and because the system returns to the same state if the loop is still intact. A restart that fixes the problem permanently is evidence that the state was purely in-memory; a restart that fixes it for minutes is evidence that the loop is still there and the restart bought only the drain time.

Cache warming belongs to the third family in a different way: pre-populating the cache before returning the instance to service avoids entering the cold-cache state at all. The cost is a slower startup and the need for a source of truth that can serve the warming reads without being saturated by them.

Detecting the state from its recovery behaviour

The diagnostic that separates a metastable failure from an ordinary overload is the response to a load reduction. If halving the traffic halves the failure rate, the system is overloaded. If halving the traffic leaves the failure rate unchanged, the system is in a state that the traffic is not causing.

The second diagnostic is the recovery time after the trigger. An overload recovers within one queue drain time, which is bounded by the queue depth divided by the excess capacity. A recovery that takes minutes, or that never completes, is not a drain.

Operationally, the signals that distinguish the two are saturation without throughput. The system’s resources are busy, its throughput is far below capacity, and the gap is accounted for by work that produces errors or by work that is duplicated. A service whose CPU is at ninety percent while its successful request rate is a third of its normal value is spending two thirds of its capacity on something other than successful requests, and the candidates are retries, error handling, and abandoned work.

The measurement problem is that the successful-request throughput is the number in the dashboard, and the work that is failing is invisible. Counting attempts separately from successes, and counting the CPU cost of the failure path, is what makes the gap visible. Queue Time, Service Time, and What a Timeout Measures covers the same asymmetry at the level of a single request.

Why the boundary is hard to find

The load at which the second equilibrium becomes reachable is not a fixed property of the system. It depends on the retry policy, on the cache hit rate, on the pool configuration, and on the latency of every dependency. Changing any of them moves the boundary, usually without anyone noticing.

The consequence is that a system can operate for months at a load that is safe under one configuration and cross into the metastable region the week a dependency gets slower or a retry policy is relaxed. The failure appears to be triggered by whatever perturbation happened at that moment, and the perturbation is only the last increment.

This is the argument for testing the loop rather than the load. A load test that ramps to the expected peak measures whether the system handles that peak in its healthy state. It does not measure whether the system can return from the failed state, and the return path is what determines whether a brief spike becomes a long outage. Injecting a transient that pushes the system into the failed state and then removing it, and observing whether the system recovers on its own, is a different test and it is the one that answers the question.

The residual role of the operator

Automated mechanisms cover the loop gain and the admission boundary. They do not cover the case where the state is held by something outside the service: a dependency that is itself metastable, a load balancer whose health checks pass for a saturated instance, or a client population that ignores Retry-After.

For those, the intervention is external and has to be decisive. Reducing traffic at the edge, disabling a retry layer, or failing over to a region with a warm cache are all changes that the system cannot make for itself. The design question is whether those levers exist and are reachable during an incident, which is a property of the deployment rather than of the code. A system that can only be recovered by a code change has a recovery time measured in deploy cycles, and a metastable failure does not wait for one.