Retry Classification When the Failure Arrives as a Timeout

An application sends a request and waits. No response arrives before its deadline. What has happened is one of four things: the request never left the device, it left and the server never received it, the server received it and its response was lost, or the server received it and is still working. From the client’s position these are indistinguishable, and they have completely different consequences for a retry.

The first two are safe to retry, because the server’s state is unchanged. The third is safe only if the server can recognise the duplicate. The fourth is not safe at all, because a retry adds load to a server already struggling and may duplicate work that will complete on its own.

A retry policy that does not distinguish these is a policy that turns an ambiguous failure into a definite one of the wrong kind.

What a timeout actually asserts

A timeout asserts that no answer arrived by the deadline. It does not assert that the request failed, and it does not even assert that the connection is broken, because a slow response and a lost connection produce the same observation at the client.

The distinction matters because the deadline is chosen by the client. A client that sets a two-second deadline on an operation whose normal duration is three seconds will time out on healthy requests, and every retry it issues will duplicate work that was about to succeed. A client that sets a sixty-second deadline on a request that normally takes fifty milliseconds will wait a minute on a dead connection before doing anything. Neither choice is a policy about failures; both are policies about the shape of the latency distribution, and the correct deadline is derived from the distribution rather than picked.

Timeouts also compose badly. If a request passes through three services each with its own deadline, and each deadline is set independently, the outermost deadline may expire before the innermost has finished. The innermost service then completes work whose result nobody will read, and the outermost retries, which starts the work again. Budgeting the deadline down the call chain, so that each layer’s deadline is smaller than its caller’s by a margin, is the standard defence; without it, retries amplify work in exactly the situation where the system can least afford it.

The transport errors that are not ambiguous

Some failures do carry information, and they should be separated from timeouts before any policy is applied.

A connection refused means the peer’s host is reachable and no process is listening on the port. The request was not processed. Retrying immediately is pointless; retrying after a delay is reasonable if the peer might be starting.

A name resolution failure means no address was obtained, so no request was sent. It is safe to retry, and it is also the failure most likely to be transient on a mobile device that has just changed networks.

A connection reset means the peer or a middlebox sent a reset. The request may or may not have been processed before the reset, so this is ambiguous in principle; in practice a reset during the request phase usually means the request was not completed, and a reset after the response began means it was.

A TLS handshake failure usually indicates a configuration problem: a certificate that does not match the name, an untrusted chain, a protocol version mismatch. Retrying does not fix configuration, and a client that retries on it turns a permanent error into a slow one. The exception is a handshake failure caused by a middlebox intercepting the connection, which is common on networks with captive portals and produces a failure that succeeds after the user authenticates.

HTTP status codes carry the server’s own classification and should be respected. A 5xx response means the server saw the request and declined or failed to complete it. Whether to retry depends on the code: 503 with a retry hint is an explicit invitation, while 500 is an unclassified failure and may be deterministic. A 4xx means the request was understood and rejected, and retrying it unchanged will be rejected again — except for the codes that specifically mean “later”, of which 408 and 429 are the ones that matter, both of which are about timing rather than content.

The Retry-After header, defined for both redirects and rate limiting, is the server’s own statement of when to come back. Honouring it is both correct and self-interested: a client that ignores it is the load the header exists to shed.

Idempotency is a property of the operation

The question a retry policy must answer is whether repeating the operation has the same effect as performing it once.

RFC 9110 defines a set of methods as idempotent: GET, HEAD, OPTIONS, TRACE, PUT, and DELETE. The definition is about the effect on the server, not about the response: repeating a DELETE may return a different status the second time, because the resource is gone, but the server’s state ends in the same place. POST is not idempotent by definition, which is why the safe default is to retry reads and to be careful with writes.

That classification is too coarse for an application, because most writes are neither purely idempotent nor purely not. Creating an order is not idempotent if the server assigns a new identifier per request, but it is idempotent if the client supplies the identifier. Incrementing a counter is not idempotent at all. Setting a field to a value is idempotent. Appending to a log is not.

The mechanism that makes a non-idempotent operation retryable is an idempotency key: a value generated by the client, sent with the request, and stored by the server alongside the result. A request carrying a key that the server has already seen returns the stored result rather than performing the operation again. The key must be generated before the first attempt and reused across every retry, which is the part implementations get wrong: a client that generates a fresh key per attempt has built a mechanism that does nothing.

The server side needs three things for this to work: a durable store for keys and results, a uniqueness constraint so two concurrent requests with the same key cannot both execute, and an expiry policy so the store does not grow without bound. The expiry is a correctness parameter, not a housekeeping detail: a key that expires while a client is still retrying allows the duplicate the mechanism was meant to prevent.

The state the server keeps

An idempotency key requires the server to remember, between two requests, that the first one happened. That memory is the cost of the guarantee, and its size is the number of in-flight or recently completed operations times the size of a stored result.

A lighter alternative is a natural key already present in the data. An operation that sets a resource to a client-chosen state, such as an address, a quantity, or a flag, is idempotent without any key, because the second application produces the same row. An operation that appends requires the client to supply a sequence number or a unique identifier per item, and the server to reject duplicates on it; that moves the storage requirement from a separate key table into the item table, which is often cheaper but requires the identifier to be part of the data model.

A third approach is to make the operation conditional. A conditional write, expressed as a precondition on a version or an entity tag, fails when the resource has changed since the client read it. A retry of such a write after a timeout will fail rather than duplicate, which is the safe direction: the client learns that its assumption no longer holds and can re-read. This is the mechanism behind optimistic concurrency, and it converts an ambiguous retry into a definite rejection.

The choice among these is a trade between storage and complexity. Keys are general and cost a table; natural keys are cheap and require designing the operation around them; preconditions are cheap and require the client to be able to re-read and retry deliberately. All three require the server to be the authority on what happened, because the client cannot be.

Deciding, with the information available

A workable policy separates decisions by what is known rather than by what is hoped. Expressed as a function over the failure, the shape is:

import socket

RETRY_LATER = {408, 429, 502, 503, 504}

def classify(exc=None, status=None, idempotent=False):
    if status is not None:
        if status in RETRY_LATER:
            return "retry" if idempotent else "retry-with-key"
        return "fail"                      # 4xx and unlisted codes
    if isinstance(exc, socket.gaierror):
        return "retry"                     # nothing was sent
    if isinstance(exc, ConnectionRefusedError):
        return "retry-later"               # peer is not listening yet
    if isinstance(exc, (TimeoutError, ConnectionResetError)):
        # ambiguous: the request may or may not have been applied
        return "retry" if idempotent else "ambiguous"
    return "fail"                          # certificate and protocol errors

The last branch is the one that requires a decision outside the function: an ambiguous outcome on a non-idempotent operation cannot be resolved by the client, and the caller must either hold the operation for the user or accept that the outcome is unknown.

Failures with a definite negative, such as resolution failure, connection refused, or a request rejected before transmission, are retried immediately, because no work was done.

Failures that are definitely permanent, such as a 4xx that is not about timing, a certificate error, or a protocol violation, are not retried, and are surfaced as errors rather than hidden behind a delay.

Timeouts and resets on idempotent operations are retried, with a delay and with a cap.

Timeouts and resets on non-idempotent operations are retried only when the request carries a key or a precondition that makes the duplicate detectable. Otherwise they are reported as ambiguous, and the application decides whether to tell the user that the outcome is unknown rather than pretending either outcome.

The delay between attempts should grow, and it should include a random component. Exponential backoff without jitter synchronises every client that failed at the same moment, so they retry at the same moment and recreate the overload that caused the failure. Jitter spreads them. The growth also gives a struggling server time to recover, which a fixed short delay does not.

The cap matters as much as the growth. A retry chain that runs for minutes is indistinguishable, from the user’s position, from a failure; and on a mobile device it competes with the radio state machine, since each attempt after the tail has expired pays a fresh connection setup. Retries that are spread over minutes are the most expensive possible pattern for the radio, which is the argument in Radio tail energy and the cost of the second request for clustering attempts rather than spacing them evenly. The other side of the same coin is the aggregate effect of many clients retrying together, which is the subject of Retry storms when a network comes back.

Where retries belong

The final question is which layer owns the retry, and the answer determines whether duplicates are even possible.

A retry at the transport layer, whether TCP’s own retransmission or QUIC’s, recovers from loss without the application knowing, and cannot duplicate an application operation, because the transport delivers a byte stream once. A retry at the application layer can duplicate an operation, and therefore needs the machinery above. A retry at an intermediary, such as a proxy, a load balancer, or a service mesh, can duplicate an operation while the client believes it sent one request, which is the most dangerous case because the client’s idempotency key, if it has one, is being reused by an intermediary that does not understand it.

The general rule is that retries should happen at the layer that can observe the failure and has the information to classify it, and that layer is usually the one closest to the operation’s semantics. Transport-level recovery is free and should be left to the transport. Application-level retries should be explicit, bounded, and conditional on the operation’s idempotency. Intermediary retries should be disabled unless the intermediary can forward the client’s key unchanged, because an intermediary that silently retries a POST is a duplicate generator with no way to detect itself.

What the client can never do is find out after the fact. The information needed to decide was lost with the response, and the only defences are making the operation safe to repeat, making the duplicate detectable, or admitting the outcome is unknown. Those three options, in that order of preference, are the whole of the design.