Hybrid Key Exchange with X25519 and ML-KEM-768

The migration to post-quantum key establishment did not wait for confidence in the new primitives. Deployed handshakes already combine an elliptic curve exchange with a lattice KEM in a single combined shared secret, so that breaking the session key requires breaking both. This hybrid construction is the pragmatic answer to a real problem: the new schemes are younger than the ones they replace, and a protocol change that could silently break every connection if a lattice assumption fails is not acceptable to operators. The interesting engineering is in what “combine” means and what an attacker who can influence one component can do to the other.

The shape of a hybrid handshake

A standard TLS 1.3 handshake derives its traffic keys from a concatenation of shared secrets, each labelled by the algorithm that produced it. The key schedule is built on HKDF, defined in RFC 5869 and used by TLS 1.3’s schedule, and the input for the early secret is a sequence of (label, secret) pairs in a defined order. A hybrid group adds a third entry: the classical ECDHE secret, the KEM shared secret, and the transcript hash up to the ServerHello. The final traffic keys are the output of the key schedule after the third entry is processed, so tampering with either component changes the result.

The composition is not a new primitive in the cryptographic sense. It is an ordered concatenation fed into a KDF, and the security property is that the result is indistinguishable from random if at least one of the components is. The formal version of that property is known as a dual-PRF or a random-oracle-secure combiner, and it has to hold not only when one component is broken but when the attacker knows the broken component entirely and chooses its value.

The handshake transcript is where the composition must be exact. The client sends its key share for the hybrid group, which contains both the X25519 public key and the ML-KEM public key concatenated in the group’s specified order. The server responds with its own share for both components. The shared secret is computed by running the classical DH with the client’s and server’s curve points, running the KEM encapsulation with the server’s KEM public key, and concatenating the two secrets. If an implementation swaps the order of the two secrets before hashing, both endpoints still derive the same key and the connection still works, but the resulting secret differs from the one the specification defines, so any future reference to the transcript or any other protocol consuming the same material diverges.

Key encapsulation is not key agreement

A KEM is a one-shot operation. The sender encapsulates to a public key and gets a ciphertext plus a shared secret; the receiver decapsulates. There is no contribution from the sender’s long-term randomness other than the ephemeral key generated for this operation. This asymmetry matters for hybrid composition because the security properties of the two halves differ. ECDHE has perfect forward secrecy from the ephemeral scalar, and the KEM inherits the same property from the ephemeral key pair generated for that handshake.

The asymmetry also matters for the failure mode. ECDHE fails only if the attacker can compute the discrete logarithm, which is a clean adversarial win. The KEM can fail incorrectly, through the decapsulation failure discussed in the ML-KEM noise walkthrough, and the implicit rejection returns a key rather than an error. In a hybrid, that behaviour is benign: a corrupted KEM secret produces a different combined key, the handshake fails at the Finished message, and the connection is torn down. The origin of the failure is not visible to either endpoint, which is the desired property.

A subtler issue is public key reuse. If the server’s KEM public key is long-term and reused across handshakes, an attacker who can distinguish the distribution of ciphertexts across encryptions has information about the key. The standard approach is to generate the KEM key pair fresh for each handshake, which is what a TLS handshake does anyway. Systems that cache KEM key pairs to avoid key generation cost, such as some SSH implementations with long-lived host keys, are choosing a different trade-off and must validate the encapsulated ciphertexts carefully.

Which attacks survive the combination and which do not

The classic argument for hybrids is that they dilute the risk of a new primitive. That argument is not automatic. If both components derive from the same randomness, or if the combiner is ambiguous, a break in one can leak the other. Three failures are worth naming.

The first is length-extension ambiguity in the combiner. If the two secrets are concatenated without a length prefix and the combiner accepts a prefix interpretation, an attacker who controls one secret can choose its length so the concatenation of a known prefix and an unknown suffix is misinterpreted. Hashing the concatenation in a domain-separated structure, with each component preceded by its identifier and length, eliminates this. NIST SP 800-56C specifies one such structure for key-derivation functions, and it treats the concatenation as a structured byte string with fixed fields.

The second is a downgrade to single-algorithm mode. The hybrid group has to be negotiated, and a server that accepts both hybrid and classical groups can be forced by a network attacker who rewrites the client’s list of supported groups to offer only the classical group. TLS 1.3 mitigates this by including the negotiated group in the transcript hash, so the Finished message covers the choice, but the attacker’s downgrade is still a successful chosen negotiation from the client’s perspective: the client sees a classical group and cannot tell whether the server made that choice. The defence is for the client to refuse classical groups entirely, not to negotiate them, which is a policy decision outside the protocol.

The third is the case where the two components are not independent. If the KEM’s encapsulation randomness is drawn from the same RNG as the ephemeral scalar, and the RNG is broken, both components degrade together. This is a threat that hybrids do not address; it is the reason implementation guidance treats the RNG as a single point of failure that has to be strong in both components.

Sizes and the client-first constraint

The bandwidth cost of the hybrid is the sum of the components plus framing. An X25519 public key is 32 bytes. An ML-KEM-768 public key is 1184 bytes and its ciphertext is 1088 bytes. A hybrid key share therefore adds roughly 1.2 KB per direction to the handshake, on top of the 32-byte ephemeral public keys. For a protocol that already carries a certificate chain of several kilobytes, this is a noticeable but tolerable increase. For a protocol used on constrained links, such as a handshake over a radio technology with a maximum transmission unit of a few hundred bytes, the increase crosses a threshold that breaks path MTU assumptions and forces fragmentation. That is a protocol design constraint, not a cryptographic one, but it is the reason device classes adopt different parameter sets.

The client-first constraint comes from the KEM’s asymmetry: the client must send a KEM public key before the server can encapsulate. In TLS 1.3, the client’s key share is already sent in the ClientHello, so the ordering fits. In protocols where the server speaks first, or where the client’s message is constrained, the hybrid has to be constructed so that the KEM public key is a server key published in advance, with the client encapsulating to it. That reverses which side sends which component and changes the transcript layout. Implementations that reuse a single “key share” structure for both directions have to get the direction flag right, because a hybrid where the client and server both send an encapsulation produces a shared secret neither side can compute.