Tolerant Parsing and Its Limits

The robustness principle, usually stated as “be conservative in what you send, be liberal in what you accept”, is the most quoted and most misapplied line in protocol design. The sending half is sound and uncontroversial. The receiving half is where the trouble is, because liberal acceptance is not a single behaviour. It is a family of behaviours with different costs, and some of them are directly harmful.

The harm is easiest to see in security. A parser that accepts several spellings of the same value gives an attacker several encodings to choose from, and any system that filters on one spelling while another component parses the other spelling has a bypass. If a URL is accepted with a backslash where the specification requires a slash, and a filter checks only the slash form, then the filter is defeated by the backslash. That class of bug exists because two parsers disagreed about what was acceptable, and the disagreement was created by tolerance.

Three kinds of tolerance, only one of which is safe

Tolerance of unknown fields is the safe kind. A receiver that skips a field it does not understand, using a length the sender supplied, does not create an alternate interpretation of anything it does understand. The message has one meaning, and the unknown field is outside it.

Tolerance of malformed encodings is the dangerous kind. A receiver that accepts a field whose length field is inconsistent with the actual bytes, or a header whose line ending is a bare LF rather than CRLF, has created a second valid encoding. Two implementations that both claim conformance may disagree about whether the message is valid, and the disagreement is exploitable.

Tolerance of semantic variation is the ambiguous middle. A receiver that accepts a value outside the specified range, such as a port number of zero or a timestamp in the past, has not created an alternate encoding but has created an alternate state. The state may be harmless, or it may bypass a check that assumed the range.

The distinctions matter because the robustness principle is usually quoted without them, and implementers apply the whole family. The modern guidance, stated in RFC 9413, is to be strict about what you send and to be predictable about what you accept: accept what the specification allows, and reject what it does not, rather than inventing a middle ground.

Where lenient parsing broke something

The record of lenient parsing is a record of interoperability failures that were resolved by tightening the specification rather than by loosening it.

HTTP header field syntax is a long example. The original grammar allowed a header value to be a sequence of tokens with specific separators, and implementations accepted values with trailing whitespace, with embedded tabs, and with obs-fold continuation lines. The obs-fold mechanism, where a header value continues onto the next line if the next line begins with whitespace, was deprecated in HTTP/1.1 and removed in HTTP/2 and HTTP/3, because it made the header block ambiguous: a proxy that joined the lines and a proxy that did not would produce different values for the same request. The removal was a deliberate narrowing, and it required senders to stop producing the construct before receivers could reject it.

The DNS name compression pointer is another. A pointer can reference an earlier offset in the message, and a lenient parser will follow a pointer to any offset, including one that points forward or that creates a loop. A strict parser requires the pointer to point backward to a name that has already been parsed, which bounds the work and prevents loops. The strict rule is not in the original specification as an explicit requirement, but it is what implementations must do to avoid unbounded recursion on a 512-byte message.

Email is the richest case, because the format has been extended so many times that the accumulated tolerance is part of the operational reality. A message with a bare LF line ending, a header with a non-ASCII byte, or an address with an unquoted comma will be accepted by most receivers and rejected by some, and the sender cannot know which. The result is that senders produce the most conservative encoding that still conveys the content, which is a form of strictness enforced by the receiver population rather than by a specification.

JSON is the counterexample where the specification chose strictness deliberately. The grammar does not permit trailing commas, comments, or single-quoted strings, and a parser that accepts them is non-conforming. The strictness was contested during the format’s standardisation and the decision held, with the argument that a small strict grammar is what makes independent implementations agree. In practice many parsers do accept extensions, and the resulting disagreements are a recurring source of bugs when data crosses between systems.

The security shape: parser differentials

The exploitable form of tolerance is a parser differential: two components in a path parse the same bytes differently, and an attacker crafts bytes that mean one thing to the filter and another to the endpoint.

The classic structure is a filter that parses conservatively and an endpoint that parses liberally. The filter sees a request for a permitted resource; the endpoint sees a request for a forbidden one. The difference is usually a character that one parser treats as a delimiter and the other treats as data: a NUL byte, a Unicode character that normalises to an ASCII delimiter, a backslash, or a semicolon.

The defence is not to make the endpoint stricter in isolation, because the endpoint’s tolerance may be needed for compatibility with real senders. The defence is to make the two parsers agree, which in practice means specifying exactly one encoding for each semantic value and rejecting everything else at the point where the message enters the trusted boundary. That is a specification decision, and it has to be made before the tolerance becomes load-bearing.

A second structure is a filter that parses liberally and an endpoint that parses conservatively. The filter permits a message that the endpoint rejects, which is usually a denial of service rather than a bypass. It is cheaper to fix because the endpoint’s rejection is visible.

The third structure is the one that is hardest to fix: two endpoints that both parse liberally but differently, with no filter in between. The disagreement surfaces as an intermittent failure, and the fix requires finding the exact input that distinguishes them. Fuzzing with differential comparison is the standard technique, and it works because the space of distinguishing inputs is usually small and structured.

The differential is also why tolerance at the framing layer is worse than tolerance at the semantic layer. A disagreement about where a field ends changes the position of every subsequent field, so the two parsers see different messages rather than different values in the same message. A disagreement about a value’s range leaves the framing intact and produces a semantic mismatch that is usually caught by a later check. Type-length-value framing reduces the first kind of disagreement to a single arithmetic check on the remaining length, which is why the check is worth stating explicitly in the specification.

Strictness has to be introduced in the right order

A specification cannot simply declare that a previously tolerated encoding is now invalid, because deployed senders still produce it. The transition has to be staged: first, senders stop producing the construct; then, receivers may warn; then, receivers may reject. Each stage requires evidence that the previous one is complete, and the evidence comes from measurement of the traffic that still contains the construct.

HTTP/2’s removal of obs-fold followed this pattern, and the specification said that a receiver must treat a message containing obs-fold as malformed, but implementations were permitted to normalise it during a transition. The permission was time-limited by the specification’s own text and by the reality that the deployed base would eventually stop producing it.

The DNS flag day efforts followed a similar pattern for EDNS handling and for the acceptance of fragmented UDP responses. The mechanism was a coordinated date after which resolvers would stop working around broken behaviour, announced in advance so that operators could fix their systems. The coordination is what made the strictness safe: without a date, each operator would wait for the others, and the broken behaviour would persist.

The general requirement is a feedback channel. A sender that produces a construct the receiver tolerates needs to learn that the tolerance is ending, and the only channels are out-of-band announcements, deprecation warnings in responses, and the eventual failure. Protocols that have no way to carry a warning make the transition harder, which is an argument for including a diagnostic channel even when the initial design does not need one.

What to be strict about

Not everything deserves strictness. The cost of rejecting a message is a broken connection, and the benefit is a smaller space of accepted inputs. The benefit is largest where the input is attacker-controlled and where a differential exists. It is smallest where the input comes from a trusted peer and where the variation is cosmetic.

The practical rule is to be strict about the fields that determine meaning and lenient about the fields that do not. A length field determines where the next field begins, so it must be exact. A field whose value is a free-form comment can accept almost anything, because no other field’s position depends on it. A field that selects a security policy must be exact, because a tolerant parse of it is a policy bypass.

Applying that rule requires knowing which fields determine meaning, and that knowledge is exactly what a specification provides. A specification that leaves a field’s grammar vague is inviting each implementation to invent its own tolerance, and the inventions will disagree. The remedy is to specify the grammar precisely and to state which deviations are errors, which is the opposite of what the robustness principle seems to recommend.

There is a version of the principle that survives this critique: be conservative in what you send, and be liberal only about the presence of things you do not need to understand. That version is compatible with strict framing and with skipping unknown extensions, and it is what the extensibility mechanisms in type-length-value framing are built to support.

Testing the boundary

A conformance suite for a strict parser has to contain both valid messages and messages that are invalid in exactly one way. The second category is the one that is usually thin, because generating an invalid message requires knowing the grammar well enough to violate it minimally, and because the expected behaviour on an invalid message is often unspecified.

The useful pattern is a mutation test: take a valid message, alter one byte, and assert that the parser either rejects the message or produces a result identical to the unaltered message. The second branch is important, because some byte alterations are in fields the parser does not care about, and a parser that rejects those is over-strict. The test distinguishes the two cases and forces the implementer to decide which is which.

A stricter variant compares two implementations on the same corpus and flags any input they classify differently. That is the differential fuzzing approach, and it finds the disagreements that matter, because a disagreement is precisely the condition an attacker needs. The cost is that it requires two implementations and a shared corpus, which is a coordination burden that most protocols do not support.

The last point is that strictness is not free at the boundary between versions. A receiver that rejects a message because of a field it does not recognise is being strict about the wrong thing, since the field’s presence is exactly what the extension mechanism permits. The line between tolerance of unknown fields and strictness about known ones is the line that separates a protocol that can change from one that cannot, and it is drawn by the criticality rules rather than by the robustness principle.