Retry Policy Design: Error Classification, Backoff, and Jitter

A retry is a bet that the failure was transient. If the bet is right, the workflow recovers without human involvement; if it is wrong, the workflow spends its time budget repeating a failure and reports it late, having possibly made the underlying condition worse. The whole of retry design is deciding which bet to place, how long to wait between attempts, and when to stop.

Failures that repeat and failures that do not

The useful split is not by subsystem but by whether a second attempt with identical inputs could plausibly succeed. Four classes cover most cases.

Transient infrastructure failures repeat successfully: a connection reset mid-transfer, a 503 from a load balancer during a rolling restart, a lock wait timeout under contention, a DNS resolution that failed because the resolver was briefly overloaded. These are the intended target of retries, and they are recognisable by their position in the stack: the failure happened below the task’s own logic.

Permanent input failures do not: a malformed record that fails parsing, a schema mismatch, a missing source file for a date that was never produced. Repeating the task with the same bytes produces the same error. These should fail immediately, because the delay before the alert is pure loss, and because a retry storm on a broken input multiplies the load on whatever is already struggling.

Resource exhaustion is ambiguous. A query rejected for exceeding a memory limit will fail identically unless the data or the cluster changed, so retrying is usually wasted; a query rejected because the warehouse’s concurrency limit was reached will succeed later, so retrying is right. The two look similar in an error message that says only “resource exceeded”. Distinguishing them requires reading the error code rather than the message text, and it is the strongest argument for classification based on structured error fields where the system provides them.

Timeouts are the hardest class. A timeout says nothing about whether the remote side completed the work, only that the local side stopped waiting. Retrying a timed-out write may duplicate the write if the first one succeeded after the timeout fired; retrying a timed-out read is safe because reads have no side effect. This asymmetry is why the idempotence of the operation, not the error class, decides whether a timeout is retryable, connecting directly to idempotent task boundaries.

A practical taxonomy tags each error with a retryable flag at the point where the error is raised, because that is the only place with enough context to know. A generic retry wrapper at the top of the task sees an exception type and a message, and cannot tell a lock timeout from a permission denial if both surface as the same exception class.

Exponential backoff and what it buys

Waiting a fixed interval between attempts synchronises the retries. If a hundred tasks fail at the same moment because a shared dependency went down, they all retry at t+5s, then at t+10s, and the dependency sees a hundred simultaneous requests each time it is already struggling. The pattern is a thundering herd, and it converts a brief outage into a longer one.

Exponential backoff spreads the attempts by increasing the interval: delay = base * 2^attempt, capped at some ceiling. With a base of one second and a cap of five minutes, the sequence is 1, 2, 4, 8, 16, 32, 64, 128, 256, 300, 300 seconds. Ten attempts cover roughly twenty-five minutes, and the cap matters because without it the eleventh wait is over eight minutes and the fifteenth is over nine hours, which no workflow wants.

The cap and the attempt count together determine the recovery window: the total wait is bounded by attempts * ceiling in the worst case, and by the sum of the geometric series in the typical case. Choosing them is choosing how long a run may spend on one task before it is declared failed, and that budget competes with the run’s own deadline. A nightly run that must finish before the morning consumers start has a fixed wall-clock budget, and a retry policy that can consume most of it is a policy that trades a chance of recovery against a certainty of lateness.

Jitter and the arrival distribution

Exponential backoff alone does not decorrelate failures that happened simultaneously, because every task computes the same delay from the same attempt number. Jitter adds randomness to the computed delay, and the variants differ in how much.

Full jitter picks a uniform random value between zero and the computed delay: sleep = random(0, min(cap, base * 2^attempt)). This produces the widest spread and the lowest peak load, at the cost of sometimes retrying almost immediately: an attempt with a computed delay of 300 seconds may sleep for one second, which wastes an attempt if the dependency is still down.

Equal jitter keeps half the delay and randomises the other half: sleep = d/2 + random(0, d/2). The minimum wait is preserved, so an attempt never fires immediately, and the spread is still substantial.

Decorrelated jitter ignores the attempt number and instead picks the next delay relative to the previous one: sleep = min(cap, random(base, prev * 3)). It adapts to the observed failure pattern rather than to a fixed schedule, and it has no natural stopping point, so it must be combined with an attempt count.

import random

def backoff_delay(attempt, base=1.0, cap=300.0, mode="full"):
    """attempt is 0-based. Returns seconds to sleep before the next try."""
    d = min(cap, base * (2 ** attempt))
    if mode == "full":
        return random.uniform(0, d)
    if mode == "equal":
        return d / 2 + random.uniform(0, d / 2)
    return d

The difference is visible in the aggregate. With a hundred tasks failing together and no jitter, each retry wave arrives as a single burst of a hundred requests. With full jitter over a 32-second window, the same hundred requests spread across the window, and the peak rate the dependency sees falls by roughly the ratio of the window to the request duration. That is the entire argument for jitter, and it holds regardless of the absolute numbers.

Retrying into an outage

The failure mode that retry policies create is amplification. A dependency that is overloaded returns errors faster than it returns successes; each error triggers a retry; the retries add load to an already overloaded system, reducing its capacity further. Under enough amplification the dependency never recovers on its own, because the retry traffic alone exceeds its capacity.

Backoff slows the amplification but does not bound it: every task in the workflow is still retrying, just less often. The bound has to come from somewhere else. Circuit breaking is the usual mechanism: after a threshold of consecutive failures, the client stops sending requests entirely for a cooldown period, failing fast instead. This converts the retry traffic to zero during the outage and lets the dependency drain its queue. The cost is that tasks fail immediately that might have succeeded, and the workflow’s error count rises during the outage even though nothing is wrong with the tasks.

Retry budgets are the alternative bound: a global allowance of retries per unit time, shared across all tasks, so that the total retry rate cannot exceed a configured fraction of the normal request rate. This preserves some retrying for the tasks that need it while preventing the aggregate from growing without limit. It requires a shared counter, which is a coordination point, which is itself a dependency that can fail.

Neither mechanism is free, and neither is present in a scheduler’s default retry setting. A per-task retry count with exponential backoff bounds each task’s attempts but not the workflow’s aggregate rate, and the aggregate is what the dependency experiences.

Interaction with the scheduler’s own timing

Retries occupy the worker slot that the task holds, in most schedulers. A task sleeping in a backoff loop is not consuming CPU but is consuming a slot in the pool, and if the pool is the constraint, a set of tasks in long backoffs can starve the rest of the graph. Schedulers that release the slot during the wait avoid this at the cost of needing to re-acquire a slot when the retry fires, which introduces queueing delay on top of the backoff.

The retry also has to interact with the run’s logical time. A task retrying past midnight for a run whose logical date is the previous day is still writing to the previous day’s partition, which is correct, but any downstream task that uses the wall clock to decide what to process will be confused. Making the logical date an explicit parameter of the task rather than something derived from the current time removes the ambiguity, and it is the reason logical dates and catchup treats the two clocks as separate quantities.

Finally, retries multiply the number of attempts the scheduler must record. A task with five retries produces up to six attempt records, each with its own start time, end time, and error. Storage for these is trivial; the difficulty is the interface that shows them, because an operator looking at a failed run needs to see the last error, not six copies of the same one, and needs to see that the failures were retried rather than that six distinct tasks failed. The scheduler’s run state has to distinguish attempts from tasks for that presentation to be possible, which is the subject of durable run state.