Retry Storms When a Network Comes Back

An outage is a period when requests fail. Recovery is the period when they all succeed at once, and it is frequently the more dangerous of the two. A network failure that affects many devices simultaneously, whether a cell site losing backhaul, a region losing a transit provider, or a certificate becoming invalid, ends at a single moment for every affected client. Each of them has been retrying, each has a queued operation or a pending screen, and each will attempt its request within the same few seconds.

The server sees a step function in load, not a ramp. If its capacity is sized for the steady state, the step exceeds it, requests time out, clients retry, and the system enters a failure mode from which the nominal recovery of the network does not release it. This is a retry storm, and the mechanisms that prevent it live on both sides of the connection.

The synchronisation that creates the storm

Retries become a storm when they are correlated, and correlation has several sources.

A fixed retry interval is the simplest and worst. Clients that fail at the same instant and wait the same duration retry at the same instant, and the server sees the same spike repeated at intervals. Every additional attempt adds to the spike rather than spreading it.

A fixed backoff with the same base and the same multiplier has the same problem in a slower form. The intervals grow, so the spikes are further apart, but each spike is still a spike and the server is still receiving the whole population in a burst.

Synchronised triggers correlate clients even without a fixed interval. A push notification sent to all devices at once causes all of them to fetch at once. A scheduled refresh at the top of the hour does the same. An application that refreshes when it returns to the foreground correlates with the user’s behaviour, which is itself correlated: everyone opens their phone at the start of the working day, and the load curve shows it.

The common structure is that the client population has a shared clock or a shared event, and any policy that maps that clock deterministically to an attempt time preserves the correlation.

Jitter, and the shapes it takes

Randomness breaks correlation, and the amount and placement of it determine how well.

The simplest form adds a random offset to the interval: an attempt after a delay drawn uniformly from a range. With a population of clients and a range comparable to the interval, the burst is spread over the range, and the peak load falls roughly in proportion to the range divided by the interval. A range that is a small fraction of the interval leaves the spike nearly intact.

A better form scales the randomness with the interval, so that later attempts are spread more widely than earlier ones. Exponential backoff with a random multiplier, drawing the delay from a range that doubles with each attempt, spreads the population progressively, which is what is wanted: the clients that failed early retry early and in a tight group, and the ones that keep failing are spread over minutes.

A third form is to randomise the moment of the first attempt rather than only the retries, which handles the correlated-trigger case. A refresh that would otherwise happen on foreground can be delayed by a random interval drawn from a distribution whose width is proportional to the freshness requirement; a screen that tolerates thirty seconds of staleness can spread its refreshes over thirty seconds.

The distinction that matters is between spreading retries and spreading the initial load. Jitter on retries prevents the storm from a shared failure. Jitter on triggers prevents the storm from a shared event. Both are needed, because they address different correlations.

A delay drawn from a range that grows with the attempt count is the form that scales with the size of the storm:

import random

def retry_delay(attempt, base=1.0, cap=60.0):
    """Full-jitter exponential backoff, in seconds."""
    ceiling = min(cap, base * (2 ** attempt))
    return random.uniform(0, ceiling)   # uniform over the whole window

Drawing from the whole window rather than from a fraction of it spreads the population as widely as the window allows, which is the behaviour that keeps a large client population from arriving together. The cap bounds the total delay, so that a client that has been failing for a long time still makes an attempt within a period the user can tolerate.

What the server can do that the client cannot

Client-side jitter is necessary and insufficient, for two reasons. It depends on every client implementing it correctly, including clients that are old versions of the application and cannot be updated; and it cannot respond to the server’s actual condition, because the client does not know it.

The server’s instruments are direct.

Refusing load is the first. A server at capacity should reject requests rather than accept them and time out, because a rejection is cheap, is fast, and carries information, while a timeout consumes a connection, a thread, and the client’s deadline. A rejection with a status that means “try later” and a hint about when is strictly better than an overloaded attempt that fails slowly. The distinction matters for the client’s classification, as described in Retry classification when the failure arrives as a timeout: a rejection is unambiguous, a timeout is not.

Telling clients when to come back is the second. The Retry-After header on a 503 or a 429 is the server’s own statement of its recovery schedule, and a client that honours it participates in the recovery rather than competing with it. Servers that omit it force clients to guess, and the guesses are wrong in the direction of optimism.

Admission control is the third, and it is the one that decides whether the recovery succeeds. The server accepts a bounded number of requests per unit time, queues a bounded number beyond that, and rejects the rest. The bound must be derived from the capacity of the slowest dependency rather than from the front end’s ability to accept connections, because accepting connections that will block on a database is how a front end appears healthy while the system is failing.

The drain rate is the fourth, and it is the one most often left to chance. If the server holds a queue of work, the rate at which it is admitted from that queue should be a configured value, not the rate at which clients happen to arrive. A queue that admits everything it receives during a storm converts the storm into a backlog whose processing time is unbounded, and the clients whose requests sit in it have long since timed out and retried, so the work is duplicated.

Prioritising what is worth recovering first

The classifications that matter are few. Work that is on a user’s critical path, meaning the request behind a screen they are looking at, is worth serving before work that is speculative. Work whose result is already stale, such as a refresh that will be superseded by the next one, is worth discarding rather than serving. Work that has been retried many times is worth deprioritising, because a client that keeps failing is often a client with a problem the server cannot solve, and serving it repeatedly consumes capacity without producing a success.

The client has a role here too. A queued write that has been waiting for an hour is more valuable than a queued write from a moment ago, because the user has been waiting longer; but a queue that drains oldest-first with equal priority will serve whatever the sequence numbers say. An outbox with a priority field, and a drain that considers it, is the mechanism, and it is a small addition to the structure described in The durable outbox.

Discarding stale work is the harder half. A client that has queued a refresh, then queued another, holds two entries that would produce the same result, and the older one is worthless. Compaction, described in the outbox article, removes it. Without compaction, a client that was offline for an hour drains an hour of redundant refreshes, and the redundancy is invisible because each entry looks like legitimate work.

The feedback that closes the loop

A storm is a positive feedback loop: overload causes failure, failure causes retry, retry causes overload. Breaking it requires a signal that reaches the client and changes its behaviour, and the available signals are limited.

The server’s rejection and its retry hint are the direct signals, and they work only for clients that read them. A client that treats every failure identically ignores the hint and retries on its own schedule, which is the behaviour the header exists to correct.

Latency is an indirect signal. A client that observes its requests taking much longer than usual can reduce its own rate, which is what congestion control does at the transport layer and what an application-level adaptive policy can do at its own layer. The difficulty is that latency is also high for reasons unrelated to server load, such as a congested cell, a distant server, or a slow radio transition, so a client that backs off on latency alone backs off on its own network conditions, which may be the correct response and may be unnecessary.

Circuit breaking is the third, and it is a client-side or intermediary-side decision to stop attempting a dependency for a period after a threshold of failures. Its value is that it converts a per-request timeout into a single fast failure for every request during the open period, which reduces both the client’s cost and the dependency’s load. Its risk is that the threshold and the open period are guesses, and a circuit that opens on a transient failure and stays open for a long period turns a blip into an outage from the client’s perspective.

The health of the loop depends on the delay between a change in server capacity and a change in client behaviour. A client with a fixed thirty-second interval takes thirty seconds to notice anything; a client that honours a retry hint notices at the speed of one response.

The pathological case: recovery that never completes

The worst outcome is a system where the network has recovered, the server has capacity, and the clients are still failing.

It happens when the backlog has grown past the point where it can be drained at the arrival rate. Suppose the server can process a thousand operations per second and the storm presents a backlog of two million, with clients retrying every thirty seconds. The backlog takes two thousand seconds to clear; during that time, clients whose operations are still queued time out and enqueue duplicates, which lengthen the backlog. The queue never empties.

Three responses exist. The server can reject the backlog outright, forcing clients to discard and resubmit with the retry hint governing the rate. That reset is brutal and effective. It can expire queued work by age, so that only recent work is attempted and the rest is answered with a failure the client can interpret. Or it can shed the duplicates by deduplicating on the idempotency key, which requires that the keys were generated at enqueue time and not at send time, as the outbox article argues.

The one that is not a response is to let the queue grow and hope. A promise that cannot be kept within the client’s patience is worse than a refusal, because it consumes capacity and produces no success.

What the client population implies for design

The final consideration is that the client population is not uniform and not under the application’s control.

Old versions of the application run retry policies that no longer match the server’s expectations. Devices with a poor connection retry more often than devices with a good one, because they fail more. Devices in a region that lost service all return at the same moment, because the network event was shared. And the population’s size is not known to the server, which sees only the requests that arrive.

A design that depends on every client behaving correctly is a design that fails on the day it is most needed. The server’s admission control must assume clients are hostile or broken and bound the damage anyway. The client’s policy must assume the server may be unavailable for a period and must be able to defer indefinitely without losing work. And the retry policy must be visible in the data the client sends: an attempt count, an enqueue timestamp, a priority, so that the server can make a decision about a request rather than about a connection.

The unifying observation is that a mobile client is a member of a large, synchronised, intermittently connected population, and every policy it adopts is multiplied by that population. A retry interval chosen for one device is a load schedule for a million, and the difference between a system that recovers and one that does not is usually a random offset applied to a timer that someone would otherwise have set to a round number.