Retry Accounting When the Scheduler Chooses the Moment

Retry logic is usually written as if the client controls when the next attempt happens. The code computes a delay, sleeps, and tries again. That model holds for a server with a thread to spare and fails completely on a phone, where the sleep itself requires a grant and the grant may arrive at a time unrelated to the delay.

Deferred retries still need backoff, but the quantity being backed off is not the same. What matters is the number of attempts the app has been permitted to make, not the number of seconds it has been waiting, and the two diverge in both directions: a device in doze can accumulate hours of nominal delay without a single attempt, and a device that wakes for other reasons can deliver three attempts inside a minute.

The attempt is the unit of cost

Every retry has two costs: the work itself, and the energy of being awake to do it. In a foreground client, the second is nearly zero because the process is already running. In a deferred client, the second dominates, because each attempt may require a wakeup that would not otherwise have happened.

That asymmetry changes what a retry budget should count. A budget of “retry for the next hour” is nearly meaningless under deferral, since an hour may contain zero attempts. A budget of “at most six attempts” is meaningful, because it bounds the number of times the device is woken and the number of times the server is hit.

The two bounds are usually combined: at most N attempts, no attempt later than some absolute deadline, and the earlier of the two wins. The deadline is what prevents a row from being retried for a week after the user has stopped caring, and the count is what prevents a tight loop of grants from hammering the server.

Backoff when the delay is not the app’s to choose

When the app sets its own delay, exponential backoff is straightforward: delay for attempt n is the base interval multiplied by a growth factor raised to n, capped at a maximum. The platform’s deferred-work schedulers do the same thing internally, with a documented backoff policy and its own minimum and maximum intervals — a minimum on the order of ten seconds and a maximum on the order of hours, and the numbers are properties of the scheduler version rather than of the app.

Layering an app-computed delay on top of the scheduler’s backoff produces the sum of two delays the app cannot see. The visible symptom is a retry that arrives far later than the app’s arithmetic predicts, and the usual reaction is to shorten the app’s delay, which changes nothing because the scheduler’s component still dominates.

The workable division of labour is to let the scheduler own the timing and to let the app own the decision. The app records the attempt and its outcome; the scheduler decides when the next run is permitted. When the outcome is retryable, the app returns a failure and lets the scheduler apply its policy. When the outcome is terminal, the app marks the row and does not ask again.

That split also handles the case the app cannot: a scheduler that has decided the device should not run anything for six hours will not run anything for six hours, and no delay computed in the app changes that.

Jitter is not optional, and it is not symmetric

Without jitter, every client that failed at the same moment retries at the same moment. The cause is usually shared — an outage, a deploy, a network partition — so the synchronised retry arrives as a second spike on a service that has just recovered.

The standard remedy is to randomise the delay. Full jitter draws the delay uniformly from zero to the computed backoff, which spreads attempts over the whole interval and produces the lowest contention. Equal jitter draws from half the backoff to the full backoff, which preserves a minimum spacing at the cost of a narrower spread.

The choice matters differently on a phone than on a server. On a phone, an attempt drawn at zero delay is an attempt that happens immediately, and if the device is currently awake, it is free. A retry storm from a million phones is not a coordinated spike in the same way a server fleet’s is, because the devices are not synchronised — their clocks differ, their grants differ, and their connectivity differs.

That said, the phone case has its own synchronisation risk: a push that wakes many devices at once, or a maintenance window that opens at a similar phase for many devices, can align attempts in a way that no jitter inside the app prevents. The defence is to apply jitter to the schedule itself, not only to the retry.

import random

def next_delay(attempt: int, base: float, cap: float) -> float:
    """Full-jitter exponential backoff.

    attempt is 1-based. The exponential term is capped before
    jitter is applied, so the distribution stays bounded by cap.
    """
    ceiling = min(cap, base * (2 ** (attempt - 1)))
    return random.uniform(0.0, ceiling)

The cap is applied to the exponential term rather than to the drawn value, which keeps the expected delay proportional to the ceiling at every attempt. Capping after the draw produces a distribution with a spike at the cap.

Idempotency is the precondition for any of this

A retry is safe only if repeating the operation has the same effect as performing it once. RFC 9110 gives the vocabulary: GET, HEAD, OPTIONS, and TRACE are defined as safe, PUT and DELETE are idempotent, and POST is neither. A retry of a POST that succeeded but whose response was lost creates a second resource.

The client-side remedies are an idempotency key — a client-generated identifier sent with the request, which the server uses to recognise a repeat — and a server-side deduplication window. Both require the server’s cooperation, and both fail if the key is regenerated on each attempt. The key must be derived from the logical operation and stored with the row, which is the same record described in Durable Deferred Work and the Persistence Boundary.

For operations that cannot be made idempotent, the only honest policy is to attempt at most once and to report the ambiguity to the user. A retry loop on a non-idempotent operation converts a network failure into duplicated data, which is a worse outcome than a visible error.

Retryable and terminal are decisions, not status codes

A five-hundred-series response is conventionally retryable and a four-hundred-series one is not, and the convention is wrong often enough to matter. A 429 is retryable and carries a Retry-After header that states how long to wait; ignoring that header and applying exponential backoff produces an attempt before the server is willing to serve it. A 408 is retryable. A 401 is terminal until the credentials are refreshed, and retrying it burns the budget without any chance of success. A 409 may be retryable after the client re-reads the current state.

The classification belongs in one place, keyed by operation, and it should distinguish three outcomes rather than two: success, retryable failure with an optional server-specified delay, and terminal failure. Collapsing the middle into either neighbour produces either a stuck row or a futile loop.

Retry-After deserves special handling because it is a server instruction that may exceed every local cap. A server that says “come back in an hour” is telling the client that a retry in thirty seconds is worse than no retry at all. Honouring it means the row’s next attempt is at least an hour away, which on a phone may mean several maintenance windows. The instruction should be stored as an absolute earliest time rather than as a delay, so that it survives a reboot.

What the maintenance window does to a deadline

A retry with an absolute deadline and a scheduler that grants one window every few hours interact badly: the deadline may fall inside a period when nothing runs, and the retry is simply skipped. From the server’s point of view the client disappeared; from the user’s, the action silently failed.

Two policies are defensible. The first extends the deadline by the observed deferral: when a row is picked up later than its scheduled time, the remaining budget is extended by the difference, so the work gets the number of attempts it was designed for. The second treats the deadline as absolute and accepts the loss, reporting it.

The first is usually right for work whose value does not expire — a log upload, a cache refresh. The second is right for work whose value does expire — a one-time code, a time-boxed offer. Which one applies is a property of the operation, and it needs to be recorded on the row rather than assumed by the retry loop.

There is a related trap with attempt counting across reboots. If attempts are counted in memory, a reboot resets the count and the row gets a fresh budget, which on a device that reboots often becomes an unbounded loop. The count belongs in the durable row, incremented at the moment the row is leased for execution rather than at the moment it succeeds.

When to stop trying altogether

A row that has exhausted its budget and a row that has failed terminally look the same from the outside, and both need to be visible. Silently deleting an exhausted row hides a persistent failure; leaving it in the queue with no attempts left wastes a scheduler slot on every pass.

The terminal state should carry the reason, and the reason should be one of a small enumerated set: the server rejected it, the local precondition no longer holds, the budget ran out, the deadline passed. That enumeration is what makes the failure diagnosable later, which is the subject of Diagnosing a Wakeup That Never Arrived.

A retry policy is also a statement about the user. Six attempts spread across a day is invisible; six attempts inside a minute, each showing a progress indicator, is a visibly broken app. The UI layer needs to know whether the current attempt is a retry and how much budget remains, or it will present a first-attempt experience on the sixth attempt.

Budgets are per operation class

One global retry budget for an app is the wrong shape, because the classes of work have different values and different costs. A message send is worth many attempts and small payloads. A photo upload is worth few attempts and large payloads. A telemetry flush is worth almost nothing and should be dropped rather than retried under constraint.

The budget therefore belongs to the operation class, and the class is a property the code knows at enqueue time. Stating it there — attempts, deadline behaviour, idempotency key, whether Retry-After overrides local caps — turns retry from a policy scattered across call sites into a field of the durable record, which is where it can be inspected after the fact.