Type-Length-Value Framing and the Extension Contract

A protocol that will never change can use fixed offsets. Every field sits at a known position, the parser reads a struct, and the format is as compact as the field widths allow. The moment a second version of that protocol has to coexist with the first, fixed offsets become a liability, because a receiver that does not know about a newly inserted field cannot find the fields after it. Type-length-value framing is the smallest change that removes this coupling. Each field carries its own identity and its own size, so a receiver that does not recognise the identity can step over the value without understanding it.

That property, skipping, is what “extensible” actually means at the wire level. It is not a property of the specification text. It is a property of the encoding.

A length field is what makes a field skippable

Consider three ways to delimit a field. The first is position: field number four begins at byte 12 because the specification says so. The second is a terminator: the field ends at the first occurrence of some byte sequence. The third is an explicit length.

Position-based framing fails the compatibility test immediately. Adding a field in the middle shifts every subsequent field; adding it at the end works only if the receiver can tell where the message ends, which usually requires an outer length that position-based designs tend to omit. Removing a field is worse, because old receivers still expect its bytes.

Terminator-based framing fails differently. A terminator can appear inside a value, so the encoding has to define an escape mechanism, and the escape mechanism has to be applied by every writer. Text protocols live with this: HTTP header fields end at CRLF, and a header value therefore cannot contain a raw CRLF. That constraint is not a flaw, but it is a constraint that propagates into every extension.

Explicit length avoids both. A TLV field is three things: an identifier drawn from a registry, a count of the bytes that follow, and those bytes. The receiver reads the identifier, reads the count, and either parses the value or adds the count to its cursor and moves on. The skipping operation requires no knowledge of the value’s internal structure, which is exactly the requirement for forward compatibility.

The identifier width determines how many kinds of field can ever exist. DHCP options use one byte for the code and one byte for the length, which gives 256 codes and a maximum option payload of 255 bytes. DHCPv6 widened the code and the length to two bytes each, giving 65,535 codes and 65,535-byte values, because DHCPv6 carries things like delegated prefixes and full option sets for relays. BGP path attributes use one byte of flags, one byte of type, and a length that is one byte by default but can be two bytes when the extended-length flag is set. TLS extensions use a two-byte type and a two-byte length, and the whole extension block is itself length-prefixed with two bytes, so the block cannot exceed 65,535 bytes.

Those numbers are not incidental. A one-byte type space is a scarce resource that will be exhausted and then argued over; a two-byte space is large enough that exhaustion is a distant problem; a variable-length integer, as QUIC uses for frame types, is effectively unbounded. Choosing the identifier width is choosing how much future negotiation the protocol will need.

Where the length stops counting, and where parsers start guessing

The subtle bugs in TLV parsing are almost never about the idea. They are about arithmetic. Three questions have to be answered unambiguously for every format: does the length include the header, does the length count bytes or elements, and is the length measured in the same units the cursor advances by.

IPv4 answers the first question with a four-bit field called IHL, which counts 32-bit words in the header including any options. A value of 5 means a 20-byte header with no options; a value of 15 means 60 bytes, the maximum. The field is four bits because the designers needed the space, and the consequence is that IPv4 options cannot exceed 40 bytes. Every option added to IPv4 competes for that fixed 40-byte budget, which is one reason IPv6 moved options into extension headers with their own length fields and an eight-octet granularity.

The second question, bytes versus elements, produces the classic off-by-one in list-valued fields. A field that carries a count of items rather than a count of bytes forces the parser to know the size of an item, which reintroduces the coupling that TLV was supposed to remove. Formats that carry lists therefore usually encode a byte length and let the receiver divide, or encode each element with its own length.

The third question is a safety issue rather than a compatibility one. A length field is attacker-controlled input, and the parser must check it against the bytes actually available before doing anything with it. The check is not just offset + length <= buffer_size, because that expression overflows when both operands are unsigned and the sum wraps. The correct form compares against the remaining space: length <= remaining. A parser that gets this wrong reads past the end of a heap allocation, and the resulting bug is remotely reachable by anyone who can send a packet.

A further check is required for containers. If a field’s value contains nested TLVs, the outer length defines a window, and the inner parse must be confined to that window rather than to the whole message. Without the confinement, a nested field can claim to extend beyond its parent, and two different parsers will disagree about where the parent ends. That disagreement is the raw material for parsing differentials, where a security device and an endpoint see different messages in the same bytes.

/* Walk a TLV buffer, skipping unknown types. Returns bytes consumed,
   or -1 if the encoding is truncated or a length overruns the parent. */
static int tlv_walk(const uint8_t *buf, size_t len, struct visitor *v)
{
    size_t off = 0;
    while (off < len) {
        if (len - off < 2)                 /* need type and length */
            return -1;
        uint8_t type = buf[off];
        uint8_t vlen = buf[off + 1];
        size_t remaining = len - off - 2;
        if (vlen > remaining)              /* compare against remaining, not off + vlen */
            return -1;
        if (!v->handle(type, buf + off + 2, vlen))
            return -1;                     /* handler rejected the value */
        off += 2 + vlen;
    }
    return (int)off;
}

The loop is nine lines and it contains the whole contract: bounded reads, a skip path that does not inspect the value, and a single exit condition. Formats that get this wrong usually get it wrong in the direction of trusting the length.

Nesting, recursion, and the cost of a container

Containers are where TLV formats earn their complexity budget and where they spend it. A container field is a TLV whose value is itself a sequence of TLVs. TLS extensions work this way: the ClientHello carries an extensions block, that block is a two-byte length followed by extensions, and each extension is a two-byte type, a two-byte length, and a value that may itself contain further length-prefixed structures. A single ClientHello can therefore be four or five levels deep before any cryptographic material is reached.

Depth costs stack. A recursive-descent parser over attacker-controlled nesting can be driven to exhaust the stack with a message that is small on the wire, because each level costs a frame while the encoding costs two bytes. The fix is either a depth limit or an explicit work queue. Both are cheap; neither is free, because the limit becomes part of the protocol’s effective specification. A sender that nests one level deeper than the limit is rejected by one implementation and accepted by another, and the difference is a compatibility failure that no specification text anticipated.

Width costs time in a different way. A flat list of N TLVs is linear to parse, but a container whose value is a list of containers that each hold a list is quadratic if the parser re-walks the outer buffer for each inner element. The bound is the same as for any nested structure: total work is proportional to the number of fields, provided each field is visited once and the offsets are carried downward rather than recomputed.

There is also an ordering question. Some formats require fields in ascending type order, which allows a parser to merge-walk two sorted lists and reject duplicates cheaply. Others allow any order and require a duplicate check that costs a hash set or a linear scan. TLS 1.3 permits extensions in any order but forbids duplicates; HTTP/2 requires that a field’s contents not depend on order. The choice matters most for the receiver’s worst case, not the sender’s average case.

What a fixed-width header can do that TLV cannot

TLV is not universally better. It costs bytes on every field, and it costs a branch in the parser for every field. A fixed header with a small number of well-chosen fields is denser and faster, and for the parts of a protocol that will never change that density is worth more than the flexibility.

The design question is therefore not “TLV or not” but “which parts of this message are allowed to grow”. IPv4 put the invariant part in a fixed 20-byte header and pushed everything optional into a bounded options area. HTTP/2 put the invariant part in a fixed nine-byte frame header (one byte of type, one of flags, three of length, four of stream identifier with one reserved bit) and pushed semantics into the frame body. TLS put the invariant part in a five-byte record header and pushed everything else into length-prefixed structures inside.

In each case the split is the same: a small fixed prefix that any version can parse, followed by self-describing content that only the interested party needs to parse. The prefix is what a middlebox or a router can read without understanding the protocol. The content is what can change. How much of a protocol should remain readable to something that is not an endpoint is the question that middlebox ossification turns on, and it is the reason the split is contested rather than obvious.

The remaining weakness of TLV is that it says nothing about semantics. A receiver can skip an unknown type safely only if the protocol defines what skipping means, and that definition is a separate design decision. Some formats treat every unknown field as ignorable; others mark fields as critical so that an unrecognised one forces rejection. That distinction, and the one bit that usually encodes it, is covered in critical bits and unknown-field handling.

A format can also fail to be extensible despite having lengths. If the length is implied by the type rather than carried explicitly, the receiver must know the type to skip the value, and the extensibility is gone. If the outer message has no total length and the transport is a byte stream, the receiver cannot find the end of the message to know that a field was truncated. If the type space is allocated first-come-first-served with no reserved range, experimenters cannot test a field without consuming a permanent number. Each of these is a small decision that closes an extension point, and they are much cheaper to get right at design time than to reopen later.