Optimistic Extension Use and the Cost of Unconfirmed Capability

A negotiation that completes before any state is committed is easy to reason about: the peer either agreed or it did not, and the code branches on the answer. The harder and more common case is a feature that is used before the peer has had a chance to disagree. The sender commits to an interpretation, the receiver may or may not support it, and the protocol has to define what happens when the support turns out to be absent.

This is optimistic use, and it appears wherever the round trip is too expensive to spend on confirmation. It is not a single mechanism but a family, and the members differ in how much state they put at risk.

Three shapes of optimism

The first shape is the harmless one: a message is sent in a form that a peer without the feature will interpret correctly under the old rules. HTTP’s optional request headers are an example. A client can send a header that asks for a feature; a server that does not know the header ignores it, and the request still means what it meant before. The feature is not delivered, and the client learns nothing, but nothing is broken.

The second shape is the detectable one: the message is sent in a form that a peer without the feature will reject, and the rejection is visible. An HTTP request with a new method, or a request that relies on a new framing, will produce an error response from a server that does not implement it. The client can then retry in the old form. The cost is one failed request, and the benefit is that the client learns the peer’s capability without a dedicated negotiation.

The third shape is the dangerous one: the message is sent in a form that a peer without the feature will accept and misinterpret. The peer believes it understood the message, acts on the old interpretation, and the sender believes the feature was used. No error is raised, and the two sides diverge. This shape is the reason optimistic use needs a specification that says exactly which forms are safe to send to an unconfirmed peer.

The design task is to make every optimistic feature fall into the first or second shape and never the third. That is a property of the encoding, not of the implementation. If the new feature’s encoding is a superset of the old one, the first shape applies. If it is disjoint and recognisable, the second applies. If it overlaps ambiguously, the third applies and the feature is not safe to use optimistically. The same judgement appears in critical-bit design, where the question is whether an uncomprehending peer can safely continue at all.

Zero round trip is the limiting case

TLS 1.3’s early data and QUIC’s zero-round-trip resumption are the most aggressive forms of optimistic use: application data is sent before the handshake completes, encrypted under keys derived from a previous connection. The peer may accept it or reject it, and the sender cannot know which until a response arrives.

The protocol’s answer to the ambiguity is to bound the risk. Early data is limited in size, the server can reject it without failing the connection, and the client is required to retry the rejected data after the handshake completes. That retry is the fallback, and it must be possible for every early-data request, which means the request must be replayable or the application must supply a token.

The state problem is the subtle part. If the server accepts early data, it has committed to a state change before it has authenticated the client’s identity in the current connection. The authentication comes from the ticket, which was issued in a previous connection, so the identity is the previous identity. A client whose credentials were revoked between the two connections can still act under the old identity until the ticket expires or the server checks.

The mitigation is a short ticket lifetime and a server-side revocation check where the risk justifies it. Both are policies, not protocol mechanisms, and the protocol’s contribution is to make the policy possible by keeping the ticket’s provenance explicit.

QUIC adds a further constraint because its zero-round-trip mode requires the client to have a valid address token, which proves the client’s address was recently reachable. The token is a defence against amplification, not against replay, and it shows that optimistic use often needs several independent checks rather than one.

The fallback path is the design

The part of an optimistic feature that gets the least attention and causes the most bugs is the path taken when the optimism fails. It has to be specified as carefully as the success path, because it runs at the worst moment: the connection is already partly established, some state has been committed, and the peer has just signalled that it cannot proceed.

There are three fallback strategies, and they differ in what they preserve.

Retry from the beginning discards all state and re-establishes with the old parameters. It is the simplest to reason about and the most expensive. A protocol that requires it has effectively made the optimistic attempt a speculative round trip, which is acceptable only if the failure is rare.

Continue with reduced function keeps the connection and drops the feature. The message that used the feature is reprocessed under the old interpretation, or the feature’s effect is undone. This is cheaper and much harder to implement, because it requires every state change caused by the feature to be reversible.

Fail the connection is the honest option when the feature cannot be undone. It converts an optimistic use into a hard requirement, which is correct if the feature was genuinely necessary and wrong if it was optional.

The choice interacts with the second shape of optimism. If the peer’s rejection is visible and specific, the sender can choose the cheapest fallback. If the rejection is a generic error, the sender cannot tell whether the feature caused it, and the only safe fallback is to retry without the feature and hope the error was unrelated. That is why a specific diagnostic is worth its cost: it turns an ambiguous failure into a recoverable one.

Caching capability to avoid repeating the guess

An optimistic use that fails is a wasted round trip. A protocol that repeats the guess on every connection wastes it repeatedly. The remedy is to record the outcome and to stop guessing for peers that have already declined.

The record is a per-peer capability cache, and it has the same problems as any cache: it can be stale, it can be poisoned, and it can be unbounded. A stale entry causes the client to skip a feature the peer now supports, which is a missed optimisation rather than a failure. A poisoned entry causes the client to skip a feature for a peer that supports it, which is the same outcome. The dangerous direction is the opposite: an entry that says the peer supports a feature it has since dropped, which causes the optimistic use to fail again.

The asymmetry suggests a policy: cache negative results aggressively and positive results conservatively. A negative result is safe because the fallback works; a positive result is a bet that the peer’s capability has not changed. Protocols that cache positive results therefore need a way to invalidate them, and the invalidation usually comes from a failed attempt.

HTTP’s Alt-Svc mechanism is a documented example of caching a capability with an expiry. A server advertises an alternative service with a lifetime, the client caches it for that lifetime, and after the lifetime the client falls back to the origin. The lifetime is the invalidation mechanism, and it is chosen by the server that knows how stable the alternative is.

What optimism does to the state machine

The implementation cost of optimistic use is a state machine with more states than the protocol’s happy path suggests. A connection that has sent early data is in a state where the data is sent but not confirmed, and the possible transitions are: confirmed, rejected, and connection failed. Each transition has to restore or discard the appropriate state, and the restore path is where the bugs live.

The common bug is a state change that is applied optimistically and not rolled back on rejection. The connection continues, the feature’s effect persists, and the peer operates under a different model. The bug is invisible in testing because the rejection path is rarely exercised, and it becomes visible only when a peer declines the feature, which is exactly the case the fallback exists for.

The structural fix is to keep optimistic state separate from confirmed state until the confirmation arrives. The feature’s effect is computed into a pending buffer, the buffer is applied on confirmation, and it is discarded on rejection. That pattern costs memory and a copy, and it makes the rollback trivial because there is nothing to roll back.

A protocol that cannot afford the copy has to make the feature’s effects reversible, which means every mutation has an inverse. That is a much stronger requirement, and it is usually only satisfiable for features that are purely local, such as HPACK’s header-compression context or a flow-control credit. Both are stateful and both can be reset, which is what makes them safe to update optimistically and roll back on rejection.

The boundary with the peer’s patience

An optimistic feature that fails repeatedly is worse than no feature, because each failure costs a round trip and the peer sees a pattern of aborted attempts. A client that tries early data on every connection to a server that never supports it pays a penalty on every connection.

The practical mitigation is to give up after a failure and to remember the failure for a period. That is the negative cache again, and its lifetime is a policy choice: too short and the client keeps retrying, too long and the client misses a capability the server has since added.

The deeper point is that optimism is a bet on the peer’s behaviour, and the bet’s expected value depends on the population of peers. A feature that most peers support is worth trying optimistically; a feature that few support is worth negotiating explicitly, because the explicit negotiation costs one round trip once and the optimistic attempt costs one round trip every time it fails. The calculus is the same one that governs negotiation round trips, applied per feature rather than per connection.

There is a version of the design that avoids the bet entirely: make the feature’s absence irrelevant by defining the old behaviour as the correct behaviour when the feature is not confirmed. That is the first shape of optimism, and it is the only shape that is safe without a fallback. Protocols that manage to express their extensions in that shape get optimistic use for free, and the ones that cannot need the machinery described here.