Cold Start After a Full Outage: Recovery Order and the Thundering Herd

Every component in a distributed system is designed to tolerate the failure of its neighbours. Almost none is designed to tolerate the simultaneous recovery of all of them. A restart is not the inverse of a crash: during the crash the system was doing no work, and during the restart every component is doing its maximum at once, against caches that are empty, connection pools that are unestablished, and dependencies that are themselves restarting. The failure modes of that state are distinct from the failure modes of steady operation, and they are not covered by the mechanisms that make the system survive an isolated failure.

What is different about the recovery state

Three properties hold during steady operation and fail during recovery.

The first is cache warmth. A service that has been running for hours answers most requests from memory, and its dependency sees a fraction of the request volume. On restart, every request misses, and the dependency sees a multiple of its steady load. That multiplier depends on the hit rate rather than on the traffic, so a service with a 95 percent hit rate presents twenty times its usual dependency load. That multiplier is invisible in capacity planning, which sizes the dependency for the steady state.

The second is that connection establishment is expensive and is normally amortised. A cold pool must open connections for each concurrent request rather than reuse them, and the handshake cost of TCP, TLS, and authentication lands on the first requests of every worker. The dependency sees a burst of connection attempts that is unrelated to the request rate and is bounded by concurrency rather than by throughput. If the dependency also limits concurrent connections per client, the attempts queue, and the queue is what the first requests wait behind.

The third is that health checks and admission decisions are made against an unrepresentative sample. A freshly started process has served no requests; its latency percentiles are computed from a handful of observations or are undefined; its error rate is zero because nothing has been attempted. A load balancer that routes on those numbers sends traffic to the least-tested instance, and a health check that passes on an empty workload passes on almost anything.

The combination produces a failure that is not attributable to any component, and it is the same feedback shape that makes a background controller overshoot when its gain is wrong. Every service is behaving as designed, and the system as a whole is failing, because the design assumed each service was surrounded by warm neighbours.

Dependency order and the cycles in it

The intuitive recovery plan is to start dependencies before dependents: databases first, then caches, then application services, then the edge. On a strictly layered system that ordering works, and it converts the recovery into a sequence of smaller recoveries, each of which can be verified before the next begins.

Most real systems are not strictly layered. A service depends on a database and on a peer service; the peer depends on the same database and on the first service for a lookup; the edge depends on both. The graph has cycles, and a cycle has no valid topological order. The practical resolution is to identify the cycles and break them at the point where a dependency can be served degraded: an authentication lookup that can fall back to a cached decision, a recommendation call that can return an empty list, a quota check that can fail open for a bounded interval. The recovery order is then a topological order of the graph after those edges are removed, with the degraded paths re-enabled only after both ends are healthy.

The other structural problem is that dependency order and resource order conflict. Starting every database before every application service is correct by dependency and wrong by resource: the database tier at full capacity with no clients is idle, while the application tier waits. Starting both together respects neither order. The compromise that works is to start the dependency tier, verify it is serving, and then start dependents in waves, which is the same wave mechanism used for rollouts, applied to a population that is currently at zero rather than at a previous version.

The herd and the ramp

When a service comes back and its clients all retry at once, the retry traffic is not the original traffic. Clients that were waiting through the outage have accumulated work, and each of them retries as soon as it can, so the instantaneous rate at recovery exceeds the steady rate by the ratio of the outage duration to the retry interval, bounded by the clients’ retry budgets but easily a large multiple.

Two mechanisms bound it, and they act at different layers.

At the client, retry budgets and jittered backoff spread the resumption over time. The jitter is the important part: a fixed backoff synchronises all clients to the same retry instant, which reproduces the herd at every retry interval. Exponential backoff with full jitter, where each delay is drawn uniformly from zero to the current backoff ceiling, decorrelates them, at the cost of some clients retrying almost immediately.

At the server, admission control bounds the rate at which work is accepted, independent of how much arrives. A queue with a bounded length and a fast rejection for overflow converts an overload into a controlled rejection rate rather than into unbounded latency. The distinction matters for recovery specifically: a server that queues everything it receives does not fail; it slows, and the slowdown propagates to every client, which then retries, which adds load. A server that rejects excess work keeps its latency bounded and lets clients learn to back off.

The ramp that combines them is progressive: accept a fraction of the target rate, hold it until the error rate and latency are stable over a window, then increase. The increment should be multiplicative at low load and additive near the target, so that the approach to capacity slows as it is neared. The shape matches a TCP congestion window, and for the same reason: the cost of overshoot near capacity is much higher than the benefit of arriving quickly.

// Ramp admits work at a rate that grows while health holds.
type Ramp struct {
    rate, target float64
    step         float64 // additive increment near target
}

func (r *Ramp) Advance(healthy bool) {
    if !healthy {
        r.rate *= 0.5 // multiplicative decrease on any breach
        return
    }
    if r.rate < 0.5*r.target {
        r.rate *= 1.5 // fast growth while far from capacity
    } else {
        r.rate += r.step // slow growth near the limit
    }
    if r.rate > r.target {
        r.rate = r.target
    }
}

Readiness is a claim about capacity, not about process state

A readiness signal is the only thing that stands between a starting process and the full weight of production traffic, and it is routinely defined too weakly. The common definition, that the process is running and its health endpoint returns success, establishes only that the process started. It does not establish that the process can serve at the rate it will be given.

A stronger definition requires evidence of capacity: the connection pool has reached its target size, the warmup queries have populated the caches that matter, the dependency clients have completed their handshakes, and the process has served a small amount of real traffic within its latency budget. Each of those is measurable, and each converts a failure that would have appeared as a latency spike into a delay in the readiness transition.

The cost of a strong readiness check is a longer recovery, because every instance now waits for its own warmup before receiving traffic. That cost is usually worth paying, and it can be reduced by warming in parallel with the ramp rather than before it: admit a small share of traffic early, use it to fill caches and pools, and increase the share as the instance demonstrates it can handle it. The two mechanisms, readiness and ramp, are the same idea at different granularities, and using both is what makes a cold start converge instead of oscillate.

There is a trap in the interaction. If the readiness check depends on a dependency that is itself cold, every instance fails readiness until the dependency warms, and the recovery stalls with no component at fault. The dependency’s own readiness must therefore be established first, which is the dependency ordering again, and the ordering has to be enforced by the recovery process rather than assumed from the startup sequence.

The state that recovery must not trust

Recovery also has to decide what to believe. On restart, a component’s view of the world is whatever it persisted, and that view may be stale in ways that steady operation would have corrected. A node that crashed while holding a lease may restart believing it still holds it; a coordinator that lost contact with its peers may restart with an outdated membership; a queue consumer may restart at an offset that was committed before its last batch was processed.

The general rule is that no component may resume acting on authority it held before the failure without re-establishing it. Leases must be reacquired, memberships re-formed, and positions re-derived from a durable record that was written atomically with the work. This is the same fencing requirement that applies to a leader that was partitioned from its peers, applied at the scale of a whole tier: the recovery is a new epoch, and nothing from the old epoch is valid until it has been re-established in the new one.

The practical form is a monotonic epoch counter, persisted and incremented at every start, that every write carries. A write bearing an older epoch than the current one is rejected by the store, which is what makes a resurrected process unable to corrupt state even if its own belief is wrong. Without such a marker, the recovery depends on every component correctly determining that it is stale, which is precisely the judgement that a crashed process cannot be trusted to make.

Recovery is a designed behaviour

The uncomfortable property of cold start is that it is rare, so it is rarely tested, and it is tested under the worst conditions: after an outage, with the people who could plan it occupied by the outage. The mechanisms that make it work are the same ones that make steady operation work, applied at a different operating point: admission control, readiness, ordering, and epoch checks. Systems that treat recovery as a designed behaviour write those mechanisms down, exercise them deliberately by restarting whole tiers during ordinary operation, and know the ramp shape and the warmup duration before they need them.

What cannot be designed away is the asymmetry between the time to fail and the time to recover. A tier can go from full service to none in seconds and take tens of minutes to return, because the return is bounded by cache warmup, connection establishment, and the deliberate slowness of the ramp. That asymmetry is the reason recovery capacity, the ability to run at reduced service while warming, matters more than the ability to restart quickly.