Retry Ladders and Dead-Letter Promotion for Failed Jobs

A background job retry and a request retry share a name and almost nothing else. A request retry has hundreds of milliseconds to work with, a caller waiting on the result, and a load-amplification problem that dominates the design. A job retry has minutes or hours, no caller waiting, and a completely different dominant cost: the job must eventually stop being retried, and the system must be able to say what happened when it did.

That difference changes the mechanism. The delay is stored in the queue as a future availability time rather than slept inside a worker, so a job waiting for its third attempt occupies no concurrency slot. The bound is a ladder of attempts with escalating gaps, and the terminal rung is a state that a human or a repair process can act on.

Three kinds of failure, three responses

Retrying by default is the most common policy and the least defensible. A failure carries information that should determine the response.

Transient failures are those where a second attempt has a genuinely different outcome distribution: a connection refused because the peer was restarting, a timeout against a dependency under load, an HTTP 429 or 503 with or without a Retry-After header. RFC 9110 makes 503 explicitly a statement that the server is unable to handle the request at the moment, which is precisely a condition that may pass.

Permanent failures are those where the input or the environment makes the outcome deterministic: a malformed payload, a referenced entity that no longer exists, a 4xx that describes the request rather than the server’s state, a permission error that a retry cannot change. Repeating these consumes attempts, delays the moment a human sees the problem, and produces a retry backlog that competes with real work.

Ambiguous failures are the interesting case: a connection reset after the request bytes were written, a timeout on a call that may have been applied, a process killed mid-execution. The handler cannot know whether the effect occurred, so the second attempt must be safe by construction — the deduplication described in Idempotency at the Effect Boundary is what makes the retry acceptable. Where the effect cannot be made safe, the correct outcome is neither a retry nor a success but a recorded ambiguity.

A job type should map its error taxonomy onto those three classes explicitly. The default for an unrecognised error is usually “ambiguous”, with a small attempt budget, rather than “transient” with a large one.

The ladder is stored state, not a sleep

The mechanism is one column. On failure, the acknowledgement writes the job back to a state where it is not runnable until a computed time:

UPDATE jobs
SET status = 'pending',
    available_at = now() + ($2 || ' seconds')::interval,
    last_error = $3,
    give_up_at = coalesce(give_up_at, now() + interval '48 hours')
WHERE id = $1 AND claim_token = $4 AND status = 'claimed';

The delay is a function of the attempt number, typically doubling from a base with a ceiling: one minute, two, four, eight, up to something like an hour, after which the interval stays flat. The base is chosen from the failure class — a dependency that is restarting recovers in seconds, a third-party service that is down recovers in hours — and the ceiling prevents a job from waiting days between attempts when its budget is measured in hours.

Sleeping inside the worker instead holds a concurrency slot, a database connection, and a lease renewal for the whole delay. With a doubling ladder reaching ten minutes, a few hundred waiting jobs saturate any fixed worker pool while doing nothing, and the lease arithmetic from Visibility Timeouts and Lease Heartbeats has to be kept alive to protect work that is not running. The stored-timestamp form costs one durable write per attempt and nothing in between.

Jitter belongs on the computed delay. Without it, every job that failed at the same moment wakes at the same moment, and a dependency that has just recovered receives the whole backlog at once — the retry storm that turns a repaired service into a broken one. Drawing the delay uniformly from a range around the computed value costs nothing and spreads the wake-ups.

The attempt counter, incremented at claim rather than at failure, is what makes the ladder terminate. It also means the ladder advances on crashes, which is what makes a crash loop terminate.

One consequence is that a deploy which stops a worker mid-job consumes an attempt even though nothing about the job failed. Releasing claims on shutdown makes the interruption cheap in time but still costs an attempt; Draining a Worker Pool Without Losing or Doubling the Work covers the sequence that keeps that cost to one attempt per interruption rather than one per lease period.

Bounds in wall-clock time, not only in attempts

An attempt limit alone is a poor bound. A job allowed twenty attempts with a delay ceiling of six hours can still be failing days after it was enqueued, long after its result stopped being useful. A report scheduled for Monday that finally succeeds on Wednesday has produced the wrong artefact.

The bound that matches the domain is a deadline recorded at enqueue: give_up_at, after which no further attempt is scheduled regardless of the attempt count. Both bounds are enforced, because they catch different pathologies. The attempt count bounds the load a single job can generate against its dependency; the wall-clock deadline bounds the staleness of a result that eventually succeeds.

The deadline should come from the caller, not from the queue. Only the caller knows whether a late result is useful, and a queue-wide default forces the wrong answer on both the job that must complete within an hour and the reconciliation that may run whenever it can. A job with no deadline still needs the attempt bound, since a job that fails deterministically with retries enabled would otherwise be retried forever.

One consequence deserves stating: a job that succeeds after its deadline is only acceptable if nothing downstream assumed it would not. Where the deadline is a promise to a user, an expired job should terminate rather than run late.

Jobs that kill their worker

A job that crashes the process — a native library fault, an out-of-memory kill, a stack exhaustion on a pathological input — never reaches its own failure path. It writes no error, schedules no retry, and does not acknowledge. The only trace is a claim whose lease expires.

If attempts are counted at failure time, that counter never moves, and the job is retried indefinitely, taking down a worker on every attempt. Counting at claim converts the crash into a consumed attempt, and after a small number of claim-expiries with no recorded error the job is promoted to a terminal state. The distinction to record is between an attempt that returned an error and an attempt that vanished; the second is a much stronger signal about the job’s input than the first.

The same mechanism catches a subtler case: a job that consistently exceeds its lease. It runs legitimately, gets reassigned, and two workers execute it concurrently until the acknowledgement is rejected. If the pattern repeats, the job’s own duration is the defect, and the ladder eventually parks it — which is the right outcome, because the alternative is a permanent duplicate-execution cycle.

Both cases argue for recording, for each claim, whether it ended in a rejection, an error, or nothing at all. A job whose attempt history is a sequence of silent expiries is diagnosable; a job whose only record is a count is not.

Dead-lettering is a state, not a destination

A dead-letter queue is often described as a separate queue that failed messages are moved to. The separate-queue form is a workaround for queue services that have no other way to express a terminal state; in a database-backed queue it is a status value plus the fields needed to act on it.

What promotion must preserve is more than the payload. The operator needs the final error, the full attempt history with timestamps, the identity of the code version that last attempted it, and the original enqueue parameters. Moving only the body of the message discards the context that makes the failure diagnosable, and the usual result is a dead-letter queue nobody reads.

Two properties make the terminal state useful. It must be excluded from the claim query — a job that is terminal but still runnable is not terminal — and it must be counted. A rate of promotion that rises after a deploy is the earliest reliable signal that the deploy broke something, and it is available without any handler-side instrumentation. The counter should be per job type, because an aggregate rate hides a single type whose failures are being absorbed by volume.

Promotion should not be silent in the other direction either. A retry policy that ends in a terminal state has decided that a unit of work will not be done; that decision belongs somewhere an owner can see it, whether the owner is a person or a repair process with its own schedule.

Re-drive without repeating the wreck

Re-driving a promoted job means enqueuing it again: a new record, or the same record with the attempt counter reset and the deadline extended. Both are correct depending on why the job failed, and neither is safe to apply to a whole dead-letter batch without review.

If the cause was a defect in the handler, re-drive reproduces the failure and refills the dead-letter state with the same payloads. If the cause was a dependency outage that has since ended, re-drive is exactly right. The distinguishing information is the error class recorded at promotion, which is another reason to store it rather than a boolean.

The mechanical hazards are throughput and duplication. Re-driving ten thousand jobs at once sends ten thousand claims at whatever the dependency is, which can re-break a service that has just recovered; rate limiting the re-drive, or staggering availability times across the batch, is the same jitter argument applied at a larger scale. And a job re-driven while a late original attempt is still in flight can execute twice, so the re-drive must either verify that the original is terminal or rely on the effect-level key to make the duplicate harmless.

A job re-driven after its input has changed is a different job. Where the payload embeds a snapshot of state taken at enqueue, re-driving replays the snapshot, which is usually what was intended; where the payload holds only an identifier and the handler reads current state, the re-drive acts on today’s data. The two behaviours are both defensible, but a re-drive procedure that does not know which one it has will produce results nobody can explain.