Dead Letter Queues and Poison Message Quarantine

From the broker’s side, a message that fails processing and a message whose consumer crashed both end up unacknowledged. The broker cannot read the error, cannot tell a malformed payload from a transient database timeout, and has no way to decide whether another attempt is worthwhile. Everything that distinguishes a poison message from an unlucky one has to be encoded by the consumer in the signals it controls: whether and how it acknowledges, how long it waits before retrying, and what it records about the attempt.

A dead letter queue is the destination for messages that have exhausted that budget. It is not a fix for failure; it is a place to put failure so that it stops consuming the capacity meant for work.

Three failures the broker cannot separate

The first is undeserializable input. A producer shipped a schema change, a payload was truncated in transit, or a field arrived with an unexpected type. No number of retries changes the bytes, so every attempt fails identically and the message is poison by construction. This is the easiest class to detect, because the failure happens before any side effect.

The second is a failed precondition. The message refers to an entity that does not exist, a state transition that is not legal from the current state, or a version that has already been superseded. The payload is valid and the handler is correct; the world is not in the shape the message assumed. Retrying can help if the missing entity is about to be created by another message, and never helps if it was deleted. Distinguishing the two requires domain knowledge, which the broker does not have.

The third is an environmental failure: the database is unreachable, the downstream API returns 503, the disk is full. These are usually transient and unrelated to the message, so immediate quarantine is the wrong response — the message is fine and the environment is not. Retrying with backoff is exactly right here, and the retry budget has to be long enough to outlast a typical outage.

The broker sees all three as a failed delivery. The consumer has to classify, and the classification determines whether the message is retried, parked, or discarded after logging.

Counting attempts is the mechanism

Since the broker cannot judge, it counts. SQS exposes an ApproximateReceiveCount attribute on every received message, incremented on each delivery. A redrive policy compares that count with maxReceiveCount and, once exceeded, moves the message to a configured dead letter queue instead of making it visible again. The attribute is named approximate because SQS does not guarantee an exact count, which is acceptable for a threshold that is meant to be comfortably above the expected retry count.

AMQP brokers use a different mechanism for the same purpose. RabbitMQ dead letters a message when it is rejected without requeue, when its per-message TTL expires, or when the queue length limit is exceeded, and it records the reason in an x-death header that accumulates entries with a count per queue and reason. A consumer can inspect that header to learn how many times the message has been dead lettered and from where, which is more information than a bare counter provides.

The common property is that the count lives outside the message body. Nothing the consumer writes into the payload is authoritative, because a failed handler may not have reached the code that would have incremented a field. Broker-side accounting survives handler crashes, which is why it is the reliable basis for a redelivery limit.

Backoff has to happen somewhere

A retry budget without delay converts a temporary outage into a fast loop. The message is received, fails, becomes visible immediately, is received again, and the consumer spends its entire capacity on the same failing work. This is the message-level version of a retry storm, and it is worse than a request-level storm because the failing unit stays in the queue rather than being dropped.

Delay can be implemented at three layers. The broker can offer a delay queue or per-message delay, as SQS does with delivery delay and as some AMQP setups do with per-message TTL plus dead lettering back to the original queue. The consumer can sleep before acknowledging a negative outcome, which occupies a worker but is simple. Or the consumer can write the message to a retry topic with a not_before timestamp and a scheduler that republishes it, which costs a second topic and a scheduler but keeps workers free.

The delay should grow with attempt number and include jitter. A deterministic exponential schedule makes every message that failed at the same moment retry at the same moment, which reproduces the original overload in synchronized waves. Jitter spreads the retries out. The base and cap are a policy choice; what matters is that the cap exists, so a message that has failed a hundred times does not wait years before its next attempt.

import random

def next_delay(attempt: int, base: float = 0.5, cap: float = 300.0) -> float:
    """Exponential backoff with full jitter, capped in seconds."""
    if attempt <= 0:
        return 0.0
    ceiling = min(cap, base * (2 ** min(attempt, 20)))
    return random.uniform(0.0, ceiling)

Full jitter draws uniformly between zero and the ceiling, spreading retries across the interval at the cost of occasionally retrying almost immediately. Decorrelated jitter, which grows from the previous delay rather than from the attempt count, is a common alternative.

The quarantine queue is a backlog with no owner

A dead letter queue looks like a normal queue and behaves like an archive of failures. The dangerous assumption is that someone will read it. Without a defined owner, a re-drive procedure, and an alert on depth, it becomes a second, invisible backlog that grows until it hits a retention limit and silently discards the evidence.

Three properties make a dead letter queue operable. Its retention must exceed the time to triage, or messages disappear before anyone looks. Its growth must be bounded and alarmed, because a sudden increase is the earliest signal of a schema change or a broken deployment. And each message must arrive with enough context to reproduce the failure: the original destination, the attempt count, the last error, and the time of the final attempt. The error text is not in the original payload, so the consumer must attach it or log it against the message id.

The queue also has to be a dead end in the topology. A common mistake is to route a dead letter queue back into the main queue on the theory that the problem will fix itself; that reconstructs the loop the dead letter queue exists to break. Re-drive should be an explicit operation, triggered by a human or by a scheduled job after a fix, not an automatic property of the queue.

A useful classification, in terms of where each failure should go:

Failure class	Retry	Destination
Undeserializable payload	Never	Dead letter immediately
Unknown schema version	Never, until a deploy	Dead letter with alert
Transient downstream error	With backoff, bounded	Back to the source queue
Business precondition	Bounded, short	Dead letter after the budget
Repeated consumer crash	Bounded by receive count	Dead letter after the budget

Skipping costs order, and that cost is real

On a partitioned log, quarantining a record is not free. The offset must advance past the failed record for the partition to make progress, so the next record is processed while the quarantined one is pending. If the two are related — an update and a delete for the same key, say — the order of effects is now wrong, and the quarantine queue has to be re-driven in a way that respects the original sequence, which is generally impossible without external coordination.

The alternatives are to stall the partition until the record is resolved, which is correct for order and unacceptable for availability, or to route the failed record and its successors for the same key aside together, which preserves per-key order at the cost of a more complex router. Systems that care about per-key ordering usually choose a bounded in-place retry and accept a temporary stall, because the stall is short and the alternative is silent reordering. That trade is the same one described in ordering guarantees and head-of-line blocking.

For lease-based queues the problem is smaller: messages are independent, so a quarantined message does not block its neighbours, and re-drive is a matter of republishing. The absence of a global order is exactly what makes this possible, which is the recurring bargain between the two models.

Bounding the path from failure to quarantine

The end-to-end guarantee a well-built system offers is narrow and worth stating precisely: every message is either processed successfully, or it reaches a quarantine destination within a bounded number of attempts and a bounded time, with its failure recorded. Nothing in that sentence requires the message to eventually succeed, and pretending otherwise produces retry loops that outlive the incident that caused them.

Two invariants keep it honest. The attempt budget must be counted by the broker, not the handler. And the final failure must be committed before the message is removed from the source, so a crash during quarantine does not lose it. Those two rules are what separate a quarantine from a discard, and they are the same ordering discipline that the acknowledgement protocols in delivery semantics impose on the success path.