AEAD Usage Limits and the KeyUpdate Handshake

A TLS connection is not limited by its keys in the way a naive model suggests. There is no nonce exhaustion at 2^32 records, because the nonce is 96 bits and the sequence number is 64 bits. The limit that actually binds is subtler: an AEAD’s authentication strength degrades as the number of records protected under one key grows, and for AES-GCM the degradation is measurable in forgery probability rather than in key recovery. TLS 1.3 turns that into a hard operational rule.

Why record count weakens authentication

AES-GCM authenticates with a polynomial evaluation over the ciphertext in GF(2^128), and the forgery probability for a single attempted forgery is bounded by roughly the number of blocks in the message divided by 2^128. Each distinct (key, nonce) pair gives an attacker one more sample of that polynomial. Over many records, the attacker accumulates samples, and the union bound over all of them is what the security proof tracks.

The consequence is that the per-record bound and the per-key bound are different quantities. A single record with a 128-bit tag is very hard to forge. A key that has protected 2^30 records gives the attacker 2^30 attempts, and an attacker willing to try all of them has a forgery probability on the order of 2^30 times the per-record bound. The tag is still 128 bits; the number of tries has grown.

RFC 8446 handles this by stating a record limit per cipher suite and requiring a sender to perform a key update before reaching it. For AES-GCM the figure is 2^24.5 full-size records, about 24 million, on a given connection, chosen to keep the authenticated-encryption advantage at roughly 2^-57. The analysis the specification points to also bounds the total plaintext under one key, and that second term is the tighter one for bulk transfer: 2^24.5 records of 16 KiB would carry roughly 2^38.5 bytes, more plaintext than the analysis permits under a single key, so a sender moving full-size fragments crosses the byte term well before it counts 24 million records. The two terms exist because the analysis tracks two quantities, the number of forgery attempts and the number of blocks processed under one key.

ChaCha20-Poly1305 has no such record-count limit in the protocol, because the sequence number would wrap long before the safety margin is reached: the counter is 64 bits, and the construction’s forgery bound does not accumulate with record count the way GCM’s does. An implementation may therefore treat the ChaCha20 limit as operationally unreachable and still enforce a key update for correctness.

The record limit is the one that binds in practice, because it counts records rather than bytes and therefore arrives sooner for a sender using small records. A connection carrying bulk data in maximum-size fragments accumulates 2^14 bytes per record, so 2^24.5 records represents a very large transfer; a connection whose records are a few hundred bytes each reaches the same count after a far smaller volume. The counter to watch is the record counter, and a sender that tracks bytes alone can cross the limit while believing it has margin.

How KeyUpdate replaces a key

KeyUpdate is a handshake message, sent inside the encrypted record layer like any other post-handshake message. It carries a single byte, request_update, which is either update_not_requested or update_requested. Receiving it causes the receiver to derive a new application traffic secret from the current one and to reset the sequence number to zero.

The derivation is one HKDF-Expand-Label call: the current application traffic secret is used as the input, the label is “traffic upd”, and the context is empty. The resulting secret yields a new key and a new IV. Because the sequence number resets, the pair of secret and counter remains unique, and the nonce construction stays sound.

def update_secret(current_secret, hash_len):
    # no transcript hash: the update is independent of handshake state
    return hkdf_expand_label(current_secret, b"traffic upd", b"", hash_len)

The empty context is what makes the update cheap: it can be performed at any time without consulting the transcript, so a sender that is running out of budget can update mid-stream without a round trip. The receiver processes the message, updates its own receive secret, and continues reading. The sender updates its send secret and continues writing.

The two directions update independently. A client updating its send key does not change the server’s send key, and the sequence numbers are per-direction, so a KeyUpdate sent by one side affects only that side’s outbound traffic. This is why the message carries a request flag rather than a mandatory action: a peer that wants the other direction refreshed can ask, and the other side may comply at its convenience.

The update must be requested, not assumed

update_requested obliges the receiver to send its own KeyUpdate before sending more application data, which is how one side can force the other to refresh. A receiver that ignores the request violates the specification, and a sender that continues under an old key after receiving a request has effectively ignored the constraint. The flag exists because the limits apply per direction, and one side may exhaust its budget long before the other.

There is a subtlety about which key protects the KeyUpdate message itself. It is sent under the old key, and the sender updates its secret only after sending it. The receiver updates after processing it. If both sides updated on sending, the message would be protected by a key the receiver had already replaced. The ordering is therefore fixed: encrypt with the current key, then update.

What goes wrong without enforcement

An implementation that ignores the limits does not fail visibly. Records continue to decrypt, connections continue to work, and the only consequence is a security margin that silently shrinks. The failure mode is an attacker who can forge a record: with enough samples under one key, the forgery probability rises from negligible to merely small, and a small probability applied to a large number of connections is a real risk. Because nothing in the protocol announces the degradation, the limit must be enforced by the sender as a matter of implementation discipline.

The counters are also easy to get wrong. The byte count should cover everything the AEAD actually encrypts, which includes the content, the inner content type byte, and any padding, and it should be counted on the ciphertext side rather than estimated from the application’s write sizes. A sender that counts application bytes only will exceed the limit by the per-record overhead, which at small record sizes is a meaningful fraction. The record count should count records sent, including empty ones, since an empty record still consumes a nonce.

TLS 1.2 had the same underlying analysis and expressed it differently. AES-GCM in TLS 1.2 used an explicit nonce, and the specification noted that the construction required care about nonce reuse but did not impose an equivalent record limit. Implementations of TLS 1.2 therefore generally did not rotate keys at all, and long-lived connections under a single key were common. The change in TLS 1.3 is not a change in the cipher; it is a decision to state the limit and provide a mechanism to respect it.

Interaction with resumption and with record padding

A key update changes only the application traffic secrets. The resumption master secret is unaffected, so a ticket issued before an update remains valid after it, and a connection that resumes from that ticket starts with a fresh sequence number under a PSK-derived key, as described in session resumption. The exporter master secret is likewise unaffected, so channel bindings computed before an update remain valid after it.

Padding interacts with the byte limit rather than the record limit. Padding bytes are inside the plaintext and are therefore encrypted and authenticated, so they count toward the byte budget. A sender that pads every record to a fixed size reaches the byte limit sooner, in proportion to how much padding it adds. This is a genuine cost of length hiding, and it is the reason padding is usually applied selectively rather than uniformly, as the analysis of record lengths sets out.

The arrangement with the fewest failure modes is for the sender to track its budget against the lower of the two limits, update at a fraction of the limit rather than at the limit itself, and treat the update as a routine event rather than an error path. A connection that transfers enough data to need several updates is not unusual; a proxy that forwards a long-lived stream may perform them regularly, and each one costs a single small record and a hash expansion, which is negligible against the traffic it protects.