ESMTP Extension Negotiation and Its Failure Modes

The original SMTP command set had no version negotiation. A client opened a session, said hello with a domain name, and proceeded through a fixed sequence of commands whose semantics could not change. Adding a capability to that protocol required either a new port or a convention in which the server’s greeting advertised what it could do. The convention won, and the mechanism it produced is a small negotiation protocol embedded in the greeting reply.

HELO, EHLO, and what the reply means

The client identifies itself with a domain name. The older form is HELO; the extended form is EHLO. The difference is not in the argument but in the reply.

A HELO reply is a single line. An EHLO reply is a multi-line reply whose first line carries the server’s identity and whose subsequent lines each carry a single extension keyword, optionally followed by parameters.

S: 220 mail.example.net ESMTP
C: EHLO relay.example.org
S: 250-mail.example.net greets relay.example.org
S: 250-PIPELINING
S: 250-SIZE 52428800
S: 250-8BITMIME
S: 250-STARTTLS
S: 250-AUTH PLAIN LOGIN
S: 250-ENHANCEDSTATUSCODES
S: 250 CHUNKING

The formatting rules are precise and are frequently violated by clients that parse the reply as text. The code 250 is followed by a hyphen for every line except the last, which is followed by a space. A client must read lines until it finds one whose fourth character is a space. A server that emits a space on a non-final line, or a hyphen on the final line, produces a reply that a strict client will misparse.

The keyword list is a statement of what the server will accept in this session. It is not a statement about the server’s general capabilities, and it is not necessarily stable: a server may advertise STARTTLS before TLS is established and stop advertising it afterwards, and it may advertise AUTH only once TLS is up. A client that caches the keyword list across sessions, or that reads it once and assumes it applies to a different connection, will attempt commands the server has stopped offering.

The absence of a keyword is meaningful. A server that does not list SIZE will not accept the SIZE parameter on MAIL FROM, and a client that sends it anyway may get a 555 or may get a silent acceptance that the server ignores. RFC 5321 specifies that a client should not use an extension the server did not advertise, which is the only thing that makes the negotiation sound.

The parameters that change the transaction

A handful of extensions alter the command sequence rather than just adding an optional field.

SIZE declares a maximum message size in octets. The value is the server’s limit, not a request. The client may include a SIZE= parameter on MAIL FROM stating the message’s size, and the server may reject the message at that point without reading the body. That early rejection is the point of the parameter: a 40 MB attachment rejected at MAIL FROM costs one command, while the same rejection after DATA costs the whole transfer.

The declaration is advisory. A server may accept a SIZE declaration and then reject the message during DATA if the actual size differs, and a client that under-declares does not gain anything. The useful discipline is to declare accurately and to treat a 552 at MAIL FROM as final for that message.

8BITMIME relaxes the octet range for the body while keeping the line-length limit. The client must not send 8-bit content unless the keyword is present, and the server must be prepared to relay or store it. The extension does not relax the line limit; that is a separate extension, and the interaction is a common source of truncated messages, as described in Header Folding and the Line Limits That Shape MIME Encoding.

PIPELINING removes the requirement to wait for a reply after each command. Without it, a session is a strict alternation: send a command, wait for the reply, send the next. With it, the client may send a group of commands (MAIL FROM, RCPT TO for each recipient, DATA) without waiting, then read the replies in order.

The saving is the round-trip time multiplied by the number of commands. On a connection with 50 ms of latency and a message with five recipients, the unpipelined session spends seven round trips after the greeting before the body starts; the pipelined session spends one. For a large queue of small messages, that difference dominates total delivery time.

The rules for pipelining are stricter than they look. The client must not pipeline a command whose reply determines whether the next command is valid. EHLO must complete before anything is sent, because the keyword list determines what may follow. DATA may be pipelined after the last RCPT TO, but the client must be prepared for the possibility that every recipient was rejected, in which case the DATA command will be answered with a 503 or 554 rather than a 354, and the client must not send the body. That case is where naive pipelining implementations fail: they send the body unconditionally and then have to abort mid-stream, leaving the server to discard a partial message.

STARTTLS must never be pipelined. The reply to STARTTLS marks the point where the connection becomes a TLS session, and any command sent before the handshake completes is either discarded or interpreted as plaintext by the wrong layer.

CHUNKING replaces the DATA command with BDAT, which carries an explicit length. BDAT n sends n octets of message content, and a final BDAT 0 LAST ends the message. The advantage is that the server knows the length in advance and never has to scan for a line containing a single period, so binary content can be transmitted without dot-stuffing and without base64 expansion. It also permits a client to stream a message it is generating rather than buffering it to compute a size.

CHUNKING is defined alongside BINARYMIME, and the pair is rarely advertised by public servers because it requires the whole delivery path to support binary content. A sender that uses it to a server that accepts it, then relays to one that does not, must convert the message, which means re-encoding the body and losing the size advantage.

Negotiation failures that matter operationally

The negotiation is simple enough that its failures are usually about state, not parsing.

A server that advertises an extension and then rejects the corresponding command is the worst case, because the client has no fallback. This happens when the keyword list is generated from a static configuration while the feature is disabled at runtime, or when a proxy in front of the server rewrites the greeting.

A client that ignores the keyword list and sends extensions anyway works against lenient servers and fails against strict ones. The failure is usually a 500 or 502 on an unrecognized command, which the client may interpret as a permanent error and bounce the message rather than retrying without the extension.

A MAIL FROM parameter that the server does not recognize must produce a 555 reply, and the client is expected to retry without the parameter. Clients that treat 555 as fatal produce avoidable bounces for messages that would have delivered on the second attempt.

The greeting itself can be the failure point. A server that takes a long time to compute its EHLO reply, because it performs a reverse DNS lookup or a policy query per connection, delays every message. The greeting is on the critical path of every session, and its cost is multiplied by the number of connections rather than the number of messages.

The EHLO argument also has a verification cost. A server that validates that the client’s claimed domain resolves to the client’s address adds a DNS lookup to every connection, and the result is only weakly informative because the claim is unauthenticated. Many servers skip the check for that reason, and the ones that perform it usually treat failure as a scoring signal rather than a rejection.

Where the extension model stops

The design has a clear boundary: extensions can add commands, add parameters to existing commands, and relax constraints, but they cannot change the meaning of a reply code or the order of the core sequence. That is why STARTTLS fits, since it adds a command and changes the connection state, and why a hypothetical extension that reordered DATA before RCPT TO would not.

The boundary is also why authentication had to be layered outside the keyword mechanism. The AUTH extension advertises mechanisms, but the decision to require authentication is a policy of the submission role, not a protocol negotiation. That split is described in The Split Between Submission and Relay Ports, and it is the reason a server can advertise AUTH and still refuse to relay for an authenticated user whose envelope sender it does not recognize.

Every extension that survived is one that a client can safely ignore. That property of graceful degradation to the base protocol is the actual design constraint, and it is what makes the keyword list a contract rather than a menu.