HTTP/1.1 Message Framing and the Cost of Connection Reuse

HTTP/1.1 has no length prefix and no type tag. Everything the parser knows comes from a header block that is itself delimited by an empty line, which means the first task on every connection is finding four bytes, CR LF CR LF, and the second is deciding whether any bytes follow that belong to the message.

This is a text protocol only in the sense that its control structures are ASCII. The body is arbitrary octets, and the only thing that separates it from the next message on a reused connection is a number or a terminator declared in the headers. Everything difficult about HTTP/1.1 parsing follows from that.

Finding the end of the header block

A request begins with a request line: method, a space, request target, a space, version, then CRLF. Header fields follow, each name: value with optional whitespace around the value, then a bare CRLF closes the block. The specification permits obsolete line folding, where a continuation line begins with a space or tab and is joined to the previous value; it is deprecated and a server may reject it, but the parser still has to decide what to do rather than treating the folded line as a malformed header name.

Two parsing rules account for most interoperability trouble. Field names are case-insensitive, so Content-Length, content-length, and CONTENT-LENGTH are the same field. Values are not, in general, but the fields that control framing take tokens and digits, which are compared without case. Whitespace before the colon is not allowed; whitespace after it is trimmed, and only spaces and tabs count as whitespace, not other control characters.

The parser must also enforce limits while it searches, because the terminator may never arrive. A server that reads until CRLF CRLF with no cap on the buffer has handed an attacker a memory exhaustion primitive; header limits and slow connections treats that failure mode in detail.

Two length fields and the rule that resolves them

Once the header block is complete, the body length comes from one of four cases.

A response to HEAD, and any response with status 1xx, 204, or 304, never has a body regardless of what the headers say. Requests have no such exemption; a request body is whatever the framing says.

If Transfer-Encoding is present and its final coding is chunked, the body is chunked. If Transfer-Encoding is present without chunked as the final coding, the body length is not knowable in advance and the message runs until the connection closes, which makes the connection unusable for anything after it.

If neither field is present, a request has no body at all. That is the common case for GET, and it is why a server can answer without waiting.

If both fields are present, the specification requires Transfer-Encoding to take precedence, and it also permits treating the combination as an error. Treating it as an error is the right choice for a server that will ever sit behind or in front of another HTTP implementation, because the two fields disagreeing is exactly the condition that request smuggling exploits: a front end that honours Content-Length forwards a request whose body it has already framed one way, and the back end, honouring Transfer-Encoding, reads the tail of that body as the start of a new request. The two hops then disagree about where messages begin, and a request the client never sent gets answered.

A Content-Length field with a value that is not a run of digits is invalid. Duplicate values are only acceptable when they are identical, and a comma-separated list of identical values is likewise acceptable; anything else is a 400. A negative value, a leading +, or a value with embedded whitespace is invalid even though some parsers historically accepted them.

def body_kind(headers):
    te = headers.get("transfer-encoding")
    cl = headers.get("content-length")
    if te is not None:
        codings = [c.strip().lower() for c in te.split(",")]
        if cl is not None:
            raise ValueError("both framing fields present")  # reject, do not resolve
        if codings[-1] != "chunked":
            raise ValueError("body delimited by connection close")
        if codings.count("chunked") != 1:
            raise ValueError("chunked applied more than once")
        return "chunked", None
    if cl is None:
        return "none", 0
    if not cl.isdigit():
        raise ValueError("non-numeric content length")
    return "fixed", int(cl)

Chunked framing on the wire

Chunked encoding turns a body of unknown length into a sequence of self-describing pieces. Each piece starts with a hexadecimal size on its own line, optionally followed by ; and chunk extensions, then CRLF, then exactly that many bytes, then CRLF. A chunk of size zero ends the body, and it may be followed by a trailer section of header fields and a final CRLF.

The motive is streaming: an origin that generates a response from a database cursor or an upstream feed cannot state a length in advance, and before chunked encoding the only option was to close the connection. Chunked keeps the connection reusable while still letting the sender start transmitting immediately.

The parser obligations are less obvious than the format. The size line is hexadecimal, so 0x10 is not valid and 10 means sixteen; leading zeros are legal but a size with more digits than the implementation accepts must be rejected rather than overflowed. Chunk extensions must be parsed and discarded, including quoted strings that may contain semicolons. A declared size larger than the remaining allowance must fail before the bytes are read, not after, or a client can announce a huge chunk and send it slowly.

Reuse, and what it changes downstream

HTTP/1.1 made persistent connections the default; Connection: close is how either side opts out. Without persistence, a connection carries one message and the close delimits the body, so framing errors are contained. With persistence, a framing error corrupts every subsequent message on that connection, and the only recovery is to close it and hope the client retries something that is safe to retry.

The performance argument for reuse is a round trip saved. A new TCP connection costs at least one round trip before any request byte moves, and on a long path that is tens of milliseconds; TLS adds its own handshake on top. Reuse also preserves congestion window state, so a connection that has been carrying traffic sends at a higher rate than one that has just completed a handshake and is still in slow start.

The cost is state. Each idle connection holds a descriptor, kernel buffers, and whatever per-connection structure the application allocated. A server therefore imposes two limits it did not need before: an idle timeout, after which the connection is closed, and often a maximum number of requests per connection, so that clients periodically rebalance across server instances instead of pinning to whichever one they reached first.

There is an unavoidable race in that design. The server decides to close an idle connection at the same moment the client sends a request on it. The client sees the connection drop after writing, and its only correct response is to retry on a new connection, which is safe for idempotent methods and questionable for others. This is why client libraries treat a connection reset on a reused connection as retryable regardless of method, an exception to the general rule that only idempotent requests may be retried automatically.

Pipelining, the technique of sending several requests before reading any response, was specified and essentially abandoned. Responses must be returned in request order, so a slow first request blocks every later one on the connection, and intermediaries in the path differed in whether they supported it. Browsers stopped using it, and the concurrency problem it was meant to solve moved to opening several connections per origin and later to multiplexing in a binary protocol.

Interim responses and protocol switches

Two framing cases break the request-response rhythm. Expect: 100-continue asks the server to send an interim 100 Continue before the client transmits a large body, so a request that will be rejected on its headers can be refused without sending the payload. A server that does not understand the expectation must still not treat the interim response as the final one; a response stream may contain any number of 1xx messages before the real one, and the body belongs only to the last.

101 Switching Protocols and the CONNECT method end HTTP framing entirely. After the switch, the connection carries an opaque byte stream, and a server that keeps trying to parse it as HTTP will corrupt the tunnel. This is the point where the connection lifecycle stops being a sequence of messages and becomes a pair of pipes, which is why upgrade handling usually sits outside the ordinary dispatch path that route matching otherwise governs.