HTTP Message Framing: Start Lines, Field Lines, and CRLF

An HTTP/1.1 message carries no length prefix and no type tag. Its shape comes from three conventions stacked on each other: two bytes that terminate every line, one empty line that ends the field section, and a framing rule that decides where the body stops. Get any of the three wrong and a parser will treat the next message on the same connection as a continuation of the current one. The grammar in RFC 9112 is written as precisely as it is because that failure is silent.

The four regions of a message

Every HTTP/1.1 message is a start line, a field section, an empty line, and optionally a body, in that order. The start line differs by direction. A request begins with a method, a single space, a request target, a single space, and the protocol version:

GET /reports/2024/q3?format=csv HTTP/1.1

A response begins with the version, a status code, and a reason phrase:

HTTP/1.1 404 Not Found

The reason phrase exists for humans reading packet captures. RFC 9110 permits it to be empty and permits clients to ignore it entirely, so no correct client distinguishes 404 Not Found from 404 or from 404 Nope. The status code is three digits; the reason phrase is arbitrary text that may contain spaces and horizontal tabs but no other control characters.

Field lines follow the start line, one per field, each of the form name, colon, optional whitespace, value, optional whitespace. The section ends at the first empty line, which is to say at the first occurrence of CRLF immediately following another CRLF. Everything after that empty line is body, and how much of it belongs to this message is the subject of message delimitation and request smuggling, where disagreements about body length become a security problem rather than a parsing inconvenience.

Why the terminator is two bytes

The line terminator is CRLF: carriage return, line feed, 0x0D 0x0A. A sender must emit both bytes. A recipient, however, may accept a bare LF as a terminator and ignore a CR that precedes it, a tolerance inherited from early implementations and kept in RFC 9112 for interoperability. That asymmetry is the first place where a parser’s behaviour depends on which role it plays. A client reading a response can be lenient; a server reading a request has to decide whether leniency is safe, because a proxy that accepts a bare LF and forwards the message with CRLF has rewritten a message it did not fully understand.

The choice of CRLF over LF alone is a historical artefact of terminal conventions rather than a protocol requirement, but it is load-bearing in one specific way: the header/body boundary is defined as the sequence CRLF CRLF. Any parser that scans for LF LF will accept messages the specification says are malformed, and will also mis-locate the boundary in a message whose field value legitimately contains a bare LF, which the grammar forbids, but which some senders produce anyway.

Whitespace has its own strictness. No space is permitted between a field name and the colon. A request containing Host : example.com must be rejected with 400, not accepted with the whitespace trimmed. The reason is that trimming is exactly the kind of leniency that lets two parsers disagree: one sees a field named Host, the other sees a field whose name ends in a space, and an intermediary forwarding the raw bytes hands the ambiguity to the next hop. Field values, by contrast, are trimmed of leading and trailing optional whitespace before interpretation, while internal whitespace is preserved.

Field names, values, and combination

A field name is a token: one or more characters from the set of alphanumerics plus !#$%&'*+-.^_|~. No spaces, no colons, no parentheses, no commas. Names are compared case-insensitively, and the canonical form in the specifications capitalises each hyphen-separated word, but content-length, Content-Length, and CONTENT-LENGTHare the same field. Field values are compared case-sensitively unless the field's own definition says otherwise;Vary: Accept-EncodingandVary: accept-encodingare the same field name inside a value, butETag: “abc”andETag: “ABC”` are different entity tags.

Multiple field lines with the same name are semantically one field whose value is the comma-separated concatenation of the individual values. A message containing

Accept: text/html
Accept: application/json

is equivalent to Accept: text/html, application/json, and a recipient may perform the combination or not, provided the result is the same. The rule has an explicit exception: fields whose grammar is not a list cannot be combined. Set-Cookie is the canonical case, since its value can contain a comma inside an expires date, so a recipient must not fold multiple Set-Cookie lines into one. A parser that combines everything uniformly will corrupt cookies.

Field ordering is not significant, with one exception worth knowing: fields with the same name must preserve their relative order when combined, because a list field may be order-sensitive in its own semantics.

Where the framing stops being obvious

Two legacy features still appear in traffic and still need a decision.

The first is obs-fold, a field line beginning with a space or horizontal tab, which continues the previous field’s value. RFC 9112 makes it obsolete: a server that receives a request containing obs-fold must either reject the message with 400 or replace each fold with one or more spaces before interpreting the value. Rejection is the safer choice, because unfolding changes the value a downstream parser would compute from the same bytes.

The second is size. There is no maximum field section size in the protocol. What the specification does recommend is that senders and recipients support URIs of at least 8000 octets, noting that on-wire representations such as the request line will necessarily be larger in some cases. Real servers impose limits, and when a limit is exceeded the correct response is 431 Request Header Fields Too Large rather than a dropped connection, so that the client can tell the difference between a rejected request and a broken network. A limit chosen too low breaks long cookies and large authorization tokens; a limit chosen too high lets an attacker spend server memory with a single connection.

Four shapes of request target

The second token of a request line is not always a path. RFC 9112 defines four forms, and which one is legal depends on the method and on whether the message is going to an origin server or a proxy.

Origin-form is the common case: an absolute path with an optional query string, starting with a slash. A request with an empty path must be normalised to / before routing, since the grammar requires at least one character. Absolute-form is a full URI including scheme and authority, and it is mandatory when the request is addressed to a proxy, because the proxy needs to know which origin to contact. Authority-form is host:port with no path at all, and it is legal only for CONNECT, whose target is the tunnel endpoint rather than a resource. Asterisk-form is a single *, legal only for OPTIONS, meaning the server itself rather than any resource on it.

The forms are not interchangeable. A server must accept absolute-form requests directly, for compatibility with clients that were configured with a proxy and later spoke to an origin, and the specification resolves the resulting conflict: the authority in the target wins, and the Host field is ignored. The rule exists so that a request cannot name one virtual host in the target and another in the field.

The Host field is mandatory in HTTP/1.1 requests; an HTTP/1.0 request whose target is absolute-form already carries the authority and has no such requirement. A message with more than one Host field is malformed. Both rules exist because the field is what selects the virtual host, and an ambiguous selector is a routing vulnerability rather than a formatting nit.

A minimal field-section reader makes the boundary explicit:

// Read until the empty line that ends the field section.
func readHead(r *bufio.Reader, max int) ([]byte, error) {
	var head []byte
	for {
		line, err := r.ReadSlice('\n')
		if err != nil {
			return nil, err // includes bufio.ErrBufferFull
		}
		if len(head)+len(line) > max {
			return nil, errTooLarge
		}
		head = append(head, line...)
		if len(line) == 2 && line[0] == '\r' {
			return head, nil // CRLF alone terminates the section
		}
	}
}

The function is deliberately intolerant: it accepts only CRLF as the terminator and returns an error on an over-long section instead of growing without bound. Both choices push the ambiguity to the caller, which is where the decision about a 400 response belongs.

HTTP/2 removed the textual framing entirely, replacing start lines and field lines with length-prefixed binary frames and a header block per stream. The semantics of the fields survived; the parsing hazards did not. That split is the useful thing to carry forward: everything in RFC 9110 about what a field means is independent of how the bytes are laid out, and the byte layout is where implementations break.