Queue Management and Retry Scheduling in MTAs

An MTA that accepts a message takes on an obligation it may not be able to discharge for days. The receiving server may be down, the destination domain’s MX may have moved, the recipient’s mailbox may be full, or the connection may fail after the message body has been sent but before the final reply arrives. The queue is the data structure that holds the obligation until it is either satisfied or provably impossible to satisfy.

The three-way split of a reply

SMTP replies carry a three-digit code. The first digit classifies the outcome, and the classification determines the queue’s action.

A 2yz reply is completion. For a message, the final 250 after DATA means the receiving server has taken responsibility. At that point the sender’s obligation ends and the queue entry can be deleted. The message may still fail later, because a receiving server can accept a message and then fail to deliver it internally, but that is the receiver’s problem, and the receiver must generate its own notification.

A 4yz reply is a transient failure. The server understood the command, could not complete it, but expects the condition to change. A full mailbox, a temporary DNS failure, a connection refused, a server that is shutting down: all of these produce 4xx, and the correct response is to keep the message and try again later. RFC 5321 is explicit that a 4xx reply to RCPT TO does not end the transaction: other recipients in the same session may still be accepted.

A 5yz reply is a permanent failure. The condition will not change by retrying: the mailbox does not exist, the domain does not accept mail, the sender is not authorized. The message for that recipient is undeliverable, and the queue must either bounce it or drop it.

The digit distinction is not the same as the enhanced status code in the reply text. A reply of 550 5.1.1 No such user carries an enhanced code whose class digit repeats the basic classification and whose subject and detail digits describe the specific condition. RFC 3463 defines that structure: class 5 permanent, 4 transient, 2 success, with subject codes grouping conditions (addressing, mailbox, network, policy, protocol) and detail codes narrowing them. A bounce message quotes the enhanced code so the sender can distinguish “no such mailbox” from “rejected by policy” without parsing prose.

The subtle case is a server that replies 5yz to a condition that is actually transient, or 4yz to one that is permanent. RFC 5321 advises that a client may treat a 4yz reply as permanent after retrying for the message’s lifetime, and that a 5yz reply must not be retried. Implementations that override this by retrying a 550 because it came from a server known to be flaky produce duplicate deliveries and backscatter.

Retry schedules and the lifetime of a message

RFC 5321 sets the queue lifetime at four to five days: a sender that has not delivered by then must give up and send a notification. It also specifies a shorter interval of four hours, after which a sender should notify the originator that delivery is delayed while continuing to retry. Those two figures, four hours and four to five days, are the only timers the base specification fixes, and they are bounds rather than a required schedule.

Within those bounds, the retry interval is a policy choice, and the shape of the curve matters more than the individual values. Retrying at a constant short interval wastes resources on a receiver that is down for hours and adds load to a server that may already be overloaded. Retrying too rarely means a transient failure that resolves in minutes takes hours to notice.

The common shape is exponential backoff with a floor and a cap: retry after a few minutes, then double, until the interval reaches some ceiling, then continue at that ceiling until the lifetime expires. The floor matters because a receiver that is restarting wants to be retried soon. The cap matters because the queue must not become a source of long-delayed duplicates: if the interval grows without bound, a message near the end of its life may be retried once and then expire.

attempt  delay        cumulative
1        0            0
2        5 min        5 min
3        10 min       15 min
4        20 min       35 min
5        40 min       1 h 15 min
6        80 min       2 h 35 min
7        160 min      5 h 15 min   (delay notification sent at 4 h)
8        320 min      10 h 35 min
...
n        cap (e.g. 4 h) until 5 days elapsed, then bounce

Two properties of this schedule deserve attention. First, the number of attempts in five days is small, on the order of thirty with a four-hour cap, so the cost of a long-lived queue entry is low and the queue can hold many of them. Second, the schedule is per recipient, not per message. A message with ten recipients where one is failing and nine succeeded keeps a queue entry only for the one, and the retry timer applies to that recipient alone.

Per-recipient queueing is a design decision with real consequences. It means the message body may be stored once and referenced by several queue entries, or copied per recipient. Storage deduplication saves space but complicates deletion: a body can only be freed when the last recipient entry referencing it is resolved. Copying per recipient is simpler and wastes space proportional to fan-out. A large mailing to a domain with one dead mailbox is the case that exposes the difference.

Notifications, and the loop they must avoid

When the lifetime expires, the sender generates a delivery status notification addressed to the envelope return path. RFC 3464 defines the format: a multipart/report with a human-readable part and a message/delivery-status part containing per-recipient fields. The per-recipient block names the final recipient, the action taken (failed), the status code, and optionally the remote MTA and the diagnostic text it returned.

The notification is itself a message, and it must not generate a notification if it fails. That is the purpose of the null return path: a notification is sent with MAIL FROM:<>, and a server that receives a message with an empty return path must never generate a bounce for it. Without that rule, two servers that both fail to deliver each other’s notifications would bounce forever.

MAIL FROM:<> 
RCPT TO:<carol@relay.example.org>
DATA
From: MAILER-DAEMON@mail.example.net
To: carol@relay.example.org
Subject: Undelivered Mail Returned to Sender
Content-Type: multipart/report; report-type=delivery-status

--boundary
Content-Type: text/plain
Delivery to the following recipient failed permanently:
    bob@example.com

--boundary
Content-Type: message/delivery-status
Reporting-MTA: dns; mail.example.net
Final-Recipient: rfc822; bob@example.com
Action: failed
Status: 5.1.1
Diagnostic-Code: smtp; 550 5.1.1 <bob@example.com>: Recipient address rejected

The Diagnostic-Code field is where the remote server’s text is preserved. It is the only part of the notification that a human can use to distinguish causes, and it is unverified: a server can return any text, including text designed to look like an instruction. That is why notifications must not be treated as a channel for commands, and why a bounce that asks the recipient to reply with credentials is a forgery.

The failure modes that make queues hard

Several problems recur across implementations.

Duplicate delivery after a lost reply. A sender transmits DATA, the receiver accepts and queues the message, and the final 250 is lost to a connection reset. The sender does not know the message was accepted, so it retries, and the receiver gets a second copy. SMTP has no transaction identifier that would let the receiver deduplicate, so the duplicate is unavoidable at the protocol level. The mitigation is at the message level: a Message-ID generated once by the originator lets the receiver drop a duplicate it has already stored, which is a heuristic and not a guarantee.

Retry storms after an outage. When a large receiver goes down, every sender’s queue starts retrying. If all senders use the same schedule with the same floor, they retry in phase and the receiver’s first minutes back in service are a synchronized flood. Jitter on the interval is the standard mitigation, and its absence is a common cause of a receiver that appears to fail repeatedly after a restart.

Queue growth from a single bad destination. A domain whose MX accepts connections but rejects every recipient with a 4xx produces queue entries that live for the full lifetime, one per message. The volume is proportional to traffic to that domain, and it persists for five days. Rate limiting per destination is the control, and it belongs in the queue, not in the SMTP client.

Backscatter. A server that accepts a message for a nonexistent recipient and then bounces it produces a notification to a return path that may be forged. The notification goes to an innocent third party. The correct behaviour is to reject at RCPT TO time, during the transaction, when the sender is still connected and can be told directly. Accepting and bouncing is the design error that produces most backscatter, and the fix is to do the recipient check before the 250 rather than after.

The queue is also where the envelope/header split becomes operational. A bounce is addressed to the envelope sender, not to the From header, which is why a message with a mismatched return path produces a notification the author never sees. That separation is described in Envelope and Header Addresses in SMTP Delivery.

The retry decision depends on the destination, and the destination is found through DNS. The lookup rules that determine which host a queue entry is retrying against are covered in MX Resolution, Preference Ordering, and Null MX.