The TLS 1.3 Handshake: What Each Flight Commits To

A full TLS 1.3 handshake for a connection with no prior state takes one round trip before the client can send application data, and the whole exchange is four messages long. That count is only meaningful because of what each message is allowed to depend on. The protocol is arranged so that the server’s entire flight is encrypted under keys the client can compute, the client’s identity check happens before any application data is accepted, and every message is bound to every message before it.

The messages are carried as records, so the encrypted flights are subject to the record layer’s framing rules, and the secrets they produce are the output of the key schedule. The certificate list inside the server’s flight is only useful once it has been turned into a path, which is the subject of certificate chain validation.

Two flights, and the line where encryption starts

The client sends a single ClientHello. The server answers with ServerHello, then EncryptedExtensions, then optionally Certificate and CertificateVerify, then Finished. The client sends Finished and may immediately follow it with application data. The server may send application data as soon as its own Finished is on the wire, without waiting for the client’s, because it has already proven its identity and the client’s Finished cannot retroactively change that.

Only two messages are in the clear: ClientHello and ServerHello. Everything after ServerHello travels under handshake traffic keys, which both sides derive from the key exchange carried in those two messages. A passive observer therefore sees the offered cipher suites, the SNI hostname, the key share groups, and the server’s selection, and nothing else. A middlebox that insists on seeing a ChangeCipherSpec message will not see one, so TLS 1.3 senders emit a single dummy ChangeCipherSpec record after ClientHello and after ServerHello, with the single byte value 0x01. It is ignored by conforming receivers and exists purely so that obsolete appliances keep forwarding the connection.

ClientHello: an offer set with one expensive guess

The message begins with a legacy version field fixed at 0x0303, a 32-byte random value, a legacy session ID, the cipher suite list, a compression list that must contain only the null method, and a block of extensions. The legacy fields are vestiges kept for compatibility with TLS 1.2 parsers.

The extensions do the real work. supported_versions lists the protocol versions offered; without it a TLS 1.3 client looks like a TLS 1.2 client and gets a TLS 1.2 handshake. signature_algorithms restricts which signature schemes the client will accept in CertificateVerify, which matters because a server holding an RSA key must know whether to sign with RSA-PSS or PKCS#1 v1.5. server_name carries the hostname the client believes it is contacting, and is visible on the wire unless a separate mechanism hides it. key_share carries one or more ephemeral public keys, each tagged with its group.

key_share is the guess. The client sends shares for the groups it expects the server to prefer, and the server may only select a group for which a share was sent. Sending several shares costs bytes; sending one risks a retry. The arithmetic is lopsided: an X25519 share is 32 bytes, while a hybrid share combining X25519 with a post-quantum key encapsulation mechanism is well over a kilobyte, because the encapsulation key alone is more than a thousand bytes. A client that offers four classical shares pays a hundred-odd bytes; one that offers four post-quantum hybrids pays several kilobytes on every connection, including the connections that fail. This is why deployment of post-quantum key exchange has been accompanied by measurements of handshake size rather than by a change in the protocol.

pre_shared_key, when present, must be the last extension in the message. The ordering is a hard requirement, not a convention: the extension carries HMAC binders computed over a truncated transcript that ends with the ClientHello itself, and the truncation point is defined as the start of the pre_shared_key extension. Anything placed after it would be outside the binder’s coverage, so nothing may be placed after it.

ServerHello: the point where keys exist

ServerHello repeats the legacy fields, echoes the session ID, names one cipher suite, and carries a small extension block: supported_versions with the selected version, key_share with the server’s public key for the selected group, and possibly pre_shared_key to indicate which offered ticket was accepted. When a PSK is selected, the server omits key_share only if it intends to skip forward secrecy entirely, which the specification allows but deployments rarely do.

Both sides can now compute the shared secret and run the key schedule. From this point the connection has handshake traffic secrets in both directions, and the server’s next flight is encrypted. The record layer takes over, which is why the framing rules and the record counter matter here: every handshake message after ServerHello is a sequence of encrypted records, and a single handshake message may span several records if it is large.

ServerHello is also where downgrade protection becomes checkable. A server that supports 1.3 but negotiates an older version sets the last eight bytes of its ServerHello random to a fixed sentinel for that version. A client that offered 1.3 checks those bytes, and if it sees the sentinel it aborts rather than continuing at the lower version. The check only works because the client knows it offered 1.3, which is one of the reasons a client should not silently retry at a lower version after a failure: a retry discards the evidence.

Certificate, CertificateVerify, and Finished

The Certificate message is a three-byte total length followed by a list of entries, each a three-byte length, a DER-encoded certificate, and an extension block. The extension block is where the server can staple an OCSP response, avoiding a separate fetch by the client. Because the lengths are three bytes, the entire message is capped at 2^24 - 1 bytes, about 16 MiB, and chains that approach that limit are a deployment problem rather than a protocol one. The server’s certificate list must be ordered so that the end-entity certificate comes first and each following certificate directly certifies the one before it. The self-signed root may be omitted, and usually is, because the client is expected to hold it already; including it costs bytes and changes nothing about the decision.

CertificateVerify proves possession of the private key for the end-entity certificate. It signs a structure built from 64 spaces, a context string naming the role and protocol version, a zero byte, and the transcript hash. The spaces and context string are a domain separation measure inherited from TLS 1.2, where the same key might be used for other signing purposes; a signature produced for one context cannot be replayed into another. The transcript hash in the signed data is what makes the signature specific to this handshake: it covers the client’s ClientHello, the server’s ServerHello, and the certificate chain itself, so an attacker who substitutes a different chain changes the hash and invalidates the signature.

Finished is the first message whose only purpose is confirmation. It carries an HMAC computed with a key derived from the sender’s handshake traffic secret, over the transcript hash of everything up to but not including the Finished itself. Verifying it proves that the peer derived the same secrets, which in turn proves that the peer saw the same transcript. A mismatch produces a decrypt_error alert rather than a partially successful connection.

The transcript hash is the spine

Every secret in the handshake is derived with the transcript hash as context. The client’s handshake traffic secret is bound to the transcript through ServerHello; the server’s application traffic secret is bound through the server’s Finished. Changing a single byte anywhere earlier changes every subsequent hash, which changes every derived key, which makes the next Finished fail. The protocol does not need a separate integrity check over the handshake because the key derivation already provides one.

This is also why the transcript is a running hash rather than a buffer. The hash state is updated as each message arrives, so memory use is constant regardless of chain size, and the cost is one hash compression per 64 or 128 bytes of handshake data. The choice of hash follows the cipher suite: SHA-256 for the AES-128-GCM and ChaCha20 suites, SHA-384 for AES-256-GCM.

When the server needs a second round trip

If the client’s key_share does not include the group the server wants, the server can reply with HelloRetryRequest instead of ServerHello. It is encoded as a ServerHello whose random field is a fixed constant, and it selects a group the client must now use. The client sends a second ClientHello, the transcript is rewound to a synthetic message_hash marker that stands in for the first one, and the handshake proceeds. The cost is one extra round trip on a connection that was supposed to take one.

Two strategies reduce the frequency of that cost, and they trade against each other. A client can send more shares, spending bytes on every connection to avoid a retry on some. A server can accept more groups, spending implementation complexity and validation effort. A third option, the cookie extension, lets a server answer a HelloRetryRequest without keeping state, which matters when the retry is caused by load balancing rather than by group mismatch. None of these removes the round trip; they only make it rarer.

The retry path has one subtle consequence for resumption. A HelloRetryRequest invalidates the early data that a client may have sent with its first flight, because the transcript the early data was bound to no longer exists. A client that sent 0-RTT data and then receives a retry must assume the data was not processed and may be replayed, which is exactly the hazard that the early data mechanism cannot rule out on its own.