SYN Cookies and the Cost of Half-Open Connections

A TCP handshake is a three-way exchange. The client sends a SYN, the server replies with a SYN-ACK, and the client sends an ACK. The connection exists from the server’s perspective only after the third segment, because until then the server has no evidence that the client’s address is real.

The natural implementation allocates a connection control block when the SYN arrives, so that the state needed to complete the handshake (the client’s initial sequence number, the negotiated options, the window scaling factor) is available when the ACK arrives. That allocation is the vulnerability. A SYN with a forged source address never produces an ACK, so the control block sits in the half-open queue until it times out, and the attacker can send SYNs faster than the queue drains.

What a half-open entry costs

Each entry in the SYN queue holds at minimum the four-tuple, the client’s ISN, the server’s ISN, the negotiated MSS and window scale, and the timestamp state. That is on the order of a hundred bytes plus whatever allocator overhead the structure carries, and the queue has a bounded length.

The bound is the crux. A queue of n entries accepts n half-open connections, and an attacker sending SYNs at any rate above the queue’s service rate fills it. Once full, the server drops new SYNs, including legitimate ones, and the service is unavailable without any need for the attacker to consume bandwidth proportional to the damage.

Raising the queue length is the obvious response and it is not sufficient, because the queue is a per-connection allocation and the number of connections is not bounded by anything the server controls. A queue large enough to absorb a sustained SYN flood is large enough to exhaust memory on its own.

Removing the state

The SYN cookie approach removes the queue. When the queue is full, or always depending on configuration, the server does not allocate a control block for the SYN. It replies with a SYN-ACK whose initial sequence number is a function of the connection parameters, and it forgets everything.

When the ACK arrives, the server recomputes the function from the values in the ACK and checks whether the result matches. If it does, the connection is established and the control block is allocated at that point, when the server has evidence that the client’s address is reachable.

The sequence number is a 32-bit value, and it must carry enough information to reconstruct the connection. The layout described in RFC 4987 splits it three ways:

| 5 bits: counter mod 32 | 3 bits: MSS index | 24 bits: keyed hash |

The counter increments once every 64 seconds, so the top five bits name one of 32 recent instants and the cookie expires by arithmetic rather than by a stored timer. The three MSS bits index into a table of eight common segment sizes, which is how the negotiated maximum segment size survives without being stored. The low 24 bits are a keyed hash over the two addresses, the two ports, the counter value, and a server secret, so an attacker who has not seen the secret cannot construct a cookie that validates.

/* Pack a cookie from the three fields. `tick` is the 64-second counter modulo 32. */
uint32_t make_cookie(uint8_t tick, uint8_t mss_index, const uint8_t hash[3])
{
    return ((uint32_t)(tick & 0x1f) << 27)      /* 5 bits of time */
         | ((uint32_t)(mss_index & 0x07) << 24) /* 3 bits of MSS index */
         | ((uint32_t)hash[0] << 16)
         | ((uint32_t)hash[1] << 8)
         |  (uint32_t)hash[2];                  /* 24 bits of keyed hash */
}

/* The completing ACK acknowledges the cookie one past its value. `h` is the hash the
   caller computed over the four-tuple and the counter value the cookie's tick names. */
int cookie_valid(uint32_t acked, uint8_t now_tick, const uint8_t h[3])
{
    uint32_t cookie = acked - 1;
    uint8_t  tick   = (uint8_t)((cookie >> 27) & 0x1f);

    /* Accept the current tick and the two preceding ones; 64 s each. */
    if (tick != now_tick && tick != (uint8_t)((now_tick - 1) & 0x1f)
        && tick != (uint8_t)((now_tick - 2) & 0x1f))
        return 0;

    /* The MSS index stays in bits 26..24; the caller reads it from the cookie. */
    return ((cookie >> 16) & 0xff) == h[0]
        && ((cookie >>  8) & 0xff) == h[1]
        && ( cookie        & 0xff) == h[2];
}

The three hash bytes come from a keyed function over the four-tuple, the counter, and a server secret. The choice of function is a deployment decision and the specification does not fix one, since only the issuing host ever verifies it; the requirement is that an attacker who has not seen the secret cannot predict the output. The appendix in RFC 4987 describes the cookie as the bitwise difference — exclusive-or — between the SYN’s sequence number and the computed value. Packing the fields directly into the sequence number, as above, validates equally well, because the client’s sequence number is visible in the completing ACK either way.

Two parameters are deliberately not encoded. The client’s initial sequence number is 32 bits of the client’s choosing, and there is no room for it alongside the hash and the counter. The server therefore acknowledges the client’s SYN with a sequence number derived from its own cookie arithmetic, and reads the client’s ISN from the sequence number of the completing ACK, less one. The larger loss is window scaling: the scale factor is a property of the receive buffer and can be recorded only by widening the MSS index or by borrowing bits from the counter. A basic cookie leaves no room for it, which caps the connection at the unscaled 65,535-byte window. The variant RFC 4987 also describes, which stores state in the timestamp value instead, exists specifically to recover those bits.

What the cookie gives up

Several costs follow, and each shows up in a different place.

Options are limited. A connection established by cookie cannot negotiate the full option set, because the server has no record of what the client offered beyond what it can encode in the MSS and window-scale index. A connection that would have used SACK and timestamps may end up without them, which degrades loss recovery on exactly the connections that are being established under load. The mechanisms lost are the ones described in SACK Blocks and the Retransmission Queue and Timestamps, RTTM, and Protection Against Wrapped Sequences.

Data carried on the SYN is lost. A client may put application data in the SYN segment itself, and a server that answers with a cookie has no state in which to queue it. RFC 4987 notes this explicitly as a limitation of the technique. Data sent in the third segment is less of a problem, because the cookie has been validated by the time the server processes it.

Servers that speak first are at risk. In protocols where the passive side sends the first application message, such as a mail server announcing readiness, the connection is not visible to the application until the handshake completes. If the client’s final ACK is lost and the server’s application never learns the connection exists, the connection fails even though the client considers it established. The failure is silent on the server side.

Replay inside the window is possible. The time counter makes a cookie valid for a bounded number of ticks, not for a single use. An attacker who can observe a SYN-ACK in flight can resend the corresponding ACK, and the server will establish a connection. The connection carries no data and will time out, but it occupies a control block, which is the resource the cookie was meant to protect. The window is what the counter buys back: with 32 tick values at 64 seconds each, a cookie issued at one tick stops validating two ticks later, so the replayable window is on the order of two minutes.

The secret must be stable. If the secret changes, every cookie in flight becomes invalid and every half-open connection is dropped. Rotating the secret requires keeping the previous value available for the acceptance window, which means holding two secrets and accepting cookies from either.

Choosing when to use cookies

Three policies are common.

Always on removes the queue entirely, so no amount of half-open connections can exhaust it. The cost is paid on every connection: the option set is limited to what the cookie encodes, and window scaling is typically lost, which caps throughput on long paths. Servers that accept a very high rate of short connections sometimes prefer this, because the queue would be a bottleneck even without an attacker.

On overflow allocates control blocks normally and switches to cookies only when the queue is full. This preserves full functionality in the common case and degrades only under attack. The risk is that the switch is triggered by the attack, so the degraded behaviour begins exactly when the server is least able to afford additional latency.

Adaptive uses a queue but drops entries from it aggressively when it fills, on the theory that the oldest entries are the least likely to complete. This does not remove the vulnerability, but it makes the queue’s failure mode less catastrophic: a full queue drops the oldest half-open connection rather than the newest SYN.

The choice is a trade between steady-state quality and worst-case survival, and it is not resolvable without knowing the traffic mix. A server whose connections are mostly long-lived bulk transfers should keep the queue, because the option negotiation matters and the connection rate is low. A server whose connections are short and numerous should use cookies always, because the queue would overflow under normal load.

The limits of the defence

Cookies defend the server’s memory, not its bandwidth. The server still emits a SYN-ACK for every SYN that arrives, so a flood costs it one outgoing segment per incoming one, plus the processing and the state the ACK later creates. Cookies do not help with that, because the server must respond to be able to complete legitimate handshakes.

The standard complement is a mechanism that filters SYNs before they reach the server, either by validating the source address at the network edge or by rate-limiting per source. Both move the decision earlier in the path, where it is cheaper. The cookie remains the last line of defence for traffic that passes the filter.

A second limit is that the cookie proves reachability of the claimed address, not identity. A distributed attack from real hosts with real addresses produces ACKs for every cookie, and the server establishes real connections at the attacker’s rate. The defence is against forged-source flooding specifically, and against that it is effective.