Nagle's Algorithm and the Delayed Acknowledgement Deadlock

Two optimisations, each sensible alone, combine into a latency trap. One batches small writes on the sender to avoid flooding the network with tiny segments. The other batches acknowledgements on the receiver to avoid flooding the network with tiny acknowledgements. Together they can produce a connection in which every request waits tens of milliseconds before it is even transmitted, and the delay is invisible in any per-operation timing because it is not caused by the network.

The small-segment problem

A TCP segment carrying one byte of payload with a 20-byte TCP header and a 20-byte IP header spends 97.5% of its bytes on overhead. Forty one-byte segments carry forty bytes of payload in 1,640 bytes of packets, and each one costs a header-processing operation at every router. The inefficiency is real, and it was severe enough in the 1980s to contribute to congestion collapse.

Nagle’s algorithm, specified in RFC 896, is a sender-side rule that prevents it. The rule: a sender may transmit a small segment, one shorter than the maximum segment size, only if it has no unacknowledged data outstanding. If there is unacknowledged data, the small segment is held until an acknowledgement arrives, at which point the held data can be sent together with any new data the application has produced in the meantime.

The effect on a bulk transfer is good. An application writing in small units accumulates data in the send buffer, and segments go out at full size. The effect on a request-response protocol is bad, because the request is exactly the case the rule holds back: the client has nothing outstanding, sends its request, and then waits. That first small write goes out immediately, so the rule does not delay the request itself. The delay appears in the second small write: the client has unacknowledged data (the request), so the next write is held until the response’s acknowledgement arrives.

This is why the classic pathology is a protocol that sends a header and then a body as two separate writes. The header goes out alone; the body waits for the header’s acknowledgement. If the peer’s acknowledgement is delayed, the body waits for the delayed acknowledgement.

The delayed acknowledgement problem

A receiver that acknowledged every segment immediately would send one acknowledgement per data segment. Those acknowledgements carry no payload, so they are pure overhead: 40 bytes of header each, plus the interrupt and processing cost at the receiver.

RFC 1122 permits a receiver to delay acknowledgements, and describes the standard policy: acknowledge every second full-sized segment, and do not delay an acknowledgement for more than about 500 ms. The delay allows a receiver to acknowledge two segments with one ACK, halving the acknowledgement traffic.

The bound is hard: the specification makes the ceiling a requirement, not a recommendation, and sets it at less than half a second. The timer itself is not fixed. Implementations choose it, and 40 ms is the value most often cited in connection with the Linux stack and with the interaction described here. It is a tuning choice, not a protocol constant, and a receiver is free to acknowledge sooner whenever the conditions above are met.

The delay is not applied to every segment. A receiver acknowledges immediately when it has a second full-sized segment to acknowledge, when it detects a gap in the sequence space (so the sender learns quickly that something is missing), or when the delayed-acknowledgement timer expires. The last case is the one that matters here.

The interaction

Put the two mechanisms on the same connection with a request-response pattern and the trap closes.

A client writes a request in two pieces: a small header, then a small body. Nagle sends the header. The body is held, because the header is unacknowledged. The server receives the header, has no second segment to acknowledge and no gap to report, and starts its delayed-acknowledgement timer. Forty milliseconds later it sends an acknowledgement. The client receives it, Nagle’s condition is satisfied, and the body is sent. The server processes the request and replies.

Every request pays the delayed-acknowledgement interval, and the interval is pure added latency. On a path whose actual round trip is 1 ms, the observed request-response time becomes 41 ms or more, and no amount of bandwidth helps.

The mirror case exists too. A server that writes a response in two pieces while the client delays its acknowledgements sees the same stall in the other direction. And the interaction compounds with a third mechanism: the delayed acknowledgement inflates the sender’s round-trip estimate, as described in Retransmission Timeout Estimation and Karn’s Algorithm, which lengthens the retransmission timer and makes any genuine loss more expensive to recover from.

/* Two ways to write the same request. The first triggers the interaction. */
write(fd, header, header_len);
write(fd, body, body_len);          /* held by Nagle until the header is ACKed */

/* The second does not: one write, one segment, nothing to hold back. */
struct iovec iov[2] = {
    { .iov_base = header, .iov_len = header_len },
    { .iov_base = body,   .iov_len = body_len   },
};
writev(fd, iov, 2);

The writev form is the correct fix at the application level, and it is preferable to disabling either mechanism. A single write produces a single segment containing both pieces, Nagle has nothing to hold back, and the receiver acknowledges it on its own schedule without the sender waiting.

Disabling the mechanisms

Both mechanisms can be disabled per socket, and the options are well known.

TCP_NODELAY disables Nagle’s algorithm on the sending socket. Small writes are transmitted immediately, regardless of unacknowledged data. The cost is exactly what Nagle was preventing: a stream of tiny segments, each with 40 bytes of header, competing for queue space with everything else on the path.

TCP_QUICKACK is a Linux socket option that requests immediate acknowledgement on the receiving socket. It is not sticky: the kernel may revert to delayed acknowledgement after the next segment, so an application that needs immediate acknowledgements must set the option repeatedly. The behaviour is a Linux extension rather than a portable interface, and code that depends on it does not port.

The usual advice is to disable Nagle only for protocols that are latency-sensitive and whose writes are already small and infrequent, and never to disable it for bulk transfer. The advice is imprecise because the real criterion is whether the application’s writes are naturally segment-sized. A protocol that writes one message per call, where messages are small, gains nothing from Nagle and loses latency to it. A protocol that streams data in arbitrary chunks gains everything from Nagle and loses nothing.

The general shape of the mistake

The interaction is an instance of a broader pattern: two independent local optimisations that each reduce a cost measured in isolation, and whose composition produces a cost neither measures. Nagle optimises bytes per segment. Delayed acknowledgement optimises acknowledgements per segment. Neither has any notion of the other’s timer, and neither is aware that the peer’s decision to wait is conditioned on the local side’s decision to wait.

The general remedy is to give the mechanisms a shared notion of time, or to remove one of them from the path. Protocols built on transports with explicit framing tend to avoid the problem entirely, because a message boundary lets the sender decide when a segment is complete rather than letting the transport infer it from the arrival pattern of writes.

There is a second-order effect worth noting: the trap is asymmetric in duration. Nagle holds data until an acknowledgement arrives, which is one round trip at minimum. Delayed acknowledgement holds the acknowledgement for a timer interval, which is independent of the round trip. On a short path the timer dominates; on a long path the round trip dominates and the timer is comparatively unimportant. This is why the pathology is most visible on low-latency local networks, where a 40 ms stall is two orders of magnitude worse than the underlying round trip, and least visible on intercontinental paths, where the round trip is already tens of milliseconds.

Where the mechanisms sit relative to the window

Neither mechanism changes how much data may be in flight. Nagle limits when a small segment may be sent, not how many bytes may be outstanding. Delayed acknowledgement limits when an acknowledgement is generated, not how much it acknowledges. Both operate within the window established by the receive window and the congestion window, and neither can violate it.

This means that a connection whose window is the binding constraint, as discussed in The Effective Window: Congestion Window, Receive Window, and Buffer Limits, does not so much avoid the stall as hide it: while the window is full the sender is blocked on an acknowledgement whatever Nagle decides about the next small write, so the two delays overlap rather than compound. The interaction proper needs spare window capacity, which the request-response pattern has by construction, because a client’s window is nearly empty when it sends.

The interaction is therefore a small-transfer problem specifically, and it is bounded by the amount of data the application writes per call rather than by anything in the network.