Store-and-Forward Delivery and the MX Indirection

The address on an envelope and the machine that receives the letter are two different things, and the postal service depends on that separation. Electronic mail inherited the separation deliberately, and the mechanism that expresses it is a single DNS record type: the mail exchanger, or MX. Everything else about internet mail, including the retry schedule, the hop-by-hop responsibility, and the fact that a message can sit in a queue for days, follows from the decision to resolve a domain to a set of candidate hosts rather than to a single destination.

Before that decision, mail transfer was a direct connection between two machines whose names appeared in the address. A sender’s host opened a connection to the recipient’s host and transferred the message. If the recipient’s host was down, the message could not be delivered, and if it was a small machine the sender had to wait for it to accept. The MX indirection introduced a level of naming between the address and the machine, and with it the ability to nominate a separate host to receive mail for a domain.

What the MX record changes

An MX record set at a domain lists one or more hosts willing to accept mail for that domain, each with a 16-bit preference value. The sending side sorts candidates by preference ascending and attempts them in order, so a domain can name a primary receiver and a backup. When two records share a preference value, the choice between them is unspecified and implementations are free to randomise, which is the only load-sharing the record type offers.

The preference field is not a priority in the queueing sense. It is an ordering of candidates for one delivery attempt, and a sender that has chosen a candidate has no obligation to fall back to a lower-preference one after the transaction has begun. A candidate that answers with a permanent refusal ends the attempt for that message, whereas one that is unreachable or answers with a temporary failure moves the sender to the next candidate.

An MX target must resolve to an address, and it must not be a CNAME. The prohibition exists because a mail exchanger’s name is used in the protocol exchange itself, in the greeting and in the receiver’s policy checks, and a canonical name that changes underneath the record would make the identity of the peer ambiguous. A target that is a CNAME is treated as a configuration error by strict implementations and tolerated by lenient ones, which is the usual shape of a rule that has no enforcement mechanism.

The null MX, an MX record with a target of a single dot and a preference of zero, is the explicit statement that a domain accepts no mail at all. It exists so that a domain can say “do not try” rather than leaving the record set empty, which under the fallback rule means “try the domain’s own address records”.

The implicit MX and its cost

When a domain has no MX records, the specification requires the sender to fall back to the domain’s address records, treating the domain name itself as the mail exchanger. This fallback keeps a large amount of legacy configuration working, and it is also a source of confusion: a domain that publishes only a web address record silently accepts mail attempts at that host, and the host may be a web server with no mail service at all.

The fallback also interacts badly with wildcard records. A wildcard address record makes every name in a zone resolve, so a typo in a domain name produces a host that answers at the IP level and refuses at the protocol level, or worse, accepts and bounces. The null MX exists precisely to make the negative statement explicit, and a domain that intends to receive no mail and does not publish one is relying on a receiver’s willingness to refuse.

Hop-by-hop responsibility

A message transferred by SMTP is accepted by one host and then becomes that host’s responsibility. The receiving server writes the message to stable storage and only then returns a success reply; if it crashes before the write completes, it must not have sent the success reply, because the sender is entitled to discard its copy once it sees that reply.

This is the opposite of the end-to-end placement used by TCP. TCP keeps a copy of every byte until the peer acknowledges it, and the application above TCP sees a reliable stream or a broken connection. SMTP terminates the TCP connection at each hop and takes ownership at the application layer, so the chain of custody is a sequence of independent transfers. A message that passes through three relays exists as three separate copies at three separate times, and each relay’s guarantee covers only its own leg.

The design reason is delay tolerance. A receiving host may be a small machine that is switched off overnight, and the sender may have a queue of thousands of messages. Holding an end-to-end connection open for hours is not viable, and retrying the whole transfer from the original sender is wasteful. Splitting the path into hops lets each hop retry independently and lets an intermediate host absorb a delay that the original sender never sees. This is a deliberate departure from the placement rule argued in The TCP/IP Split and the End-to-End Argument: mail moves reliability to the hops precisely because the endpoints cannot be assumed to be available at the same time.

S: 220 mail.example.net ESMTP
C: EHLO sender.example.org
S: 250-mail.example.net
S: 250 SIZE <maximum message size in bytes>
C: MAIL FROM:<alice@sender.example.org> SIZE=<message size>
S: 250 2.1.0 Ok
C: RCPT TO:<bob@example.net>
S: 250 2.1.5 Ok
C: DATA
S: 354 End data with <CR><LF>.<CR><LF>
C: ...message body, dot-stuffed...
C: .
S: 250 2.0.0 Ok: queued as <queue identifier>

The reply code at the end of DATA is the ownership transfer. Everything before it is negotiation that may be abandoned without consequence; after it, the sender may delete its copy and the receiver has committed to delivering or to generating a bounce.

The dot-stuffing rule and why framing needs an escape

The message body is terminated by a line containing a single period. That means a body line consisting of a single period would terminate the message early, so the sender prefixes an extra period to any line that begins with one, and the receiver removes exactly one leading period from any line that has one. The transformation is not reversible in general if applied twice, which is why it is defined as a transfer encoding applied once at the sender and undone once at the receiver, not as a property of the stored message.

The same problem appears at a smaller scale with the line ending. SMTP uses CRLF as the line terminator, and a body containing a bare LF must be normalised before transmission. The normalisation is another transformation that changes the bytes, so a checksum computed over the stored message will not match one computed over the transmitted form.

The rules interact with the maximum line length: a line in the message body may not exceed 998 characters plus the CRLF, and a receiver must be able to accept at least that. A message with a longer line must be wrapped or encoded, and the encoding that does so is part of the message format rather than the transfer protocol.

What the retry rules actually require

A temporary failure is signalled by a 4xx reply, and the sender must retry later rather than bounce. The specification sets the expectation that a sender will keep trying for several days before giving up, and that the interval between attempts will grow. It does not fix a schedule, and the practical schedules differ widely, which is why a message can appear to be lost for a day and then arrive.

A permanent failure is a 5xx reply, and the sender must not retry it. The distinction matters because the receiver’s state may change: a 550 for a nonexistent mailbox is permanent for that transaction, while a 451 for a temporary local problem invites a retry. A receiver that returns 5xx for a condition that is actually transient causes the sender to discard a deliverable message, and a receiver that returns 4xx for a condition that will never change causes the sender to keep a queue entry alive for days.

def classify(reply_code):
    """SMTP reply classes, per RFC 5321 section 4.2.1."""
    if 200 <= reply_code < 300:
        return "success"
    if 300 <= reply_code < 400:
        return "intermediate"       # e.g. 354, continue with the data
    if 400 <= reply_code < 500:
        return "transient"          # queue and retry
    if 500 <= reply_code < 600:
        return "permanent"          # bounce, do not retry
    raise ValueError("not a valid SMTP reply code")

The three-digit code is structured: the first digit is the class, the second is the subject category, and the third refines it. A receiver is required to send a code whose class matches the condition, and a sender is required to act on the class rather than on the text. The text after the code is for humans and must not be parsed, which is a rule that has been violated by nearly every implementation that wanted to distinguish two conditions sharing a code.

Where the model strains

Store-and-forward assumes that an intermediate host is willing to hold a message it cannot deliver and to keep trying. That assumption is what spam filtering and abuse control attacked: an open relay that accepts mail for any domain and retries indefinitely is a resource for sending unsolicited mail, and the response was to restrict relaying to known parties. The restriction changes the topology from a mesh of willing forwarders to a set of domains that accept mail only for their own users, and a sender that is not authorised for a domain is refused at the first hop.

The second strain is authentication of the sender. The protocol has no mechanism to prove that the party in MAIL FROM is entitled to use that domain, so the identity in the envelope is a claim. Later additions attach signatures and publish policy records in DNS, and they work by giving the receiver a way to check the claim rather than by changing the transfer model.

The third strain is the queue itself. A host that accepts a message and then cannot deliver it for days must keep it on disk, and the size of that store is a resource an attacker can exhaust by sending many messages to addresses that will never resolve. The mitigation is to refuse at the protocol level rather than accept and queue, which is why a receiver that cannot verify a recipient at the time of the transaction is in a weaker position than one that can.

The dependency on DNS is the same one described in DNS Delegation, Glue Records, and the Referral Loop: the MX lookup can fail, and a failure to resolve is a temporary condition that must be retried rather than treated as a permanent rejection. A resolver outage therefore becomes a mail delay, which is a coupling that the design accepts deliberately because the alternative is a hard failure at the moment of sending.