TCP Keepalive as a Failure Detector, and Why It Cannot Be One

A long-lived connection carries no traffic for hours and then has to be used. Whether the peer at the other end still exists is a question the connection state cannot answer, because TCP keeps no timer for idleness by default. The keepalive option exists to answer it, and the mismatch between what it answers and what a failover system needs is the reason almost every distributed protocol reinvents liveness at the application layer.

What the probe is, at the packet level

A keepalive probe is a segment with the sequence number set to one less than the next expected sequence number, carrying no data. It is deliberately outside the window of acceptable data: the receiver must respond with an acknowledgement of its current expected sequence number, and that acknowledgement, carrying no data, does not advance anything. The trick is that the probe is unacknowledgeable as data, so it cannot corrupt the stream, and any response at all proves the peer’s TCP stack is processing segments.

Three parameters govern it. The idle time is how long the connection must be quiet before the first probe. The interval is the spacing between probes once started. The count is how many unanswered probes constitute failure. The failure is reported to the application as an error on the socket, typically ETIMEDOUT on the next operation, and the connection is torn down.

The defaults are the problem. RFC 1122 requires that the default idle interval be at least two hours, and the common Unix default matches that floor, with a probe interval of 75 seconds and nine unanswered probes before the connection is abandoned. Two hours plus nine probes at 75 seconds is a detection window of well over two hours. For a system that needs to notice a dead peer within seconds, that is not a detector, it is a garbage collector for abandoned connections.

Why the two-hour default is defensible

The default is not an oversight. Keepalive traffic is generated by the kernel on behalf of an application that did not ask for it, and every probe is a packet that a peer’s stack must process. On a host with tens of thousands of idle connections, enabling keepalive with a short idle time turns an idle server into a packet generator, and the cost is paid by the peer, which may be a busy server whose only sin was accepting a connection.

The second reason is that keepalive was designed to reap dead connections, not to detect failure. A connection whose peer crashed without closing leaves the local socket in ESTABLISHED forever, consuming a file descriptor and a slot in the connection table. A two-hour idle time reclaims that descriptor eventually, which is all the original purpose required. Nothing in that purpose needs the detection to be fast.

The third reason is that a short idle time changes the meaning of an idle connection. A connection used once an hour, with a keepalive idle time of five minutes, becomes a connection that sends probes for most of its life. The probes are cheap individually and not cheap in aggregate.

What a successful probe proves, and what it does not

A keepalive acknowledgement proves that the peer’s TCP stack received a segment and produced a response. It does not prove that the peer’s application is alive, that it is not deadlocked, that it is not in a stop-the-world pause, or that it can service a request. A process whose event loop is blocked still has a kernel that acknowledges segments, because acknowledgement is the kernel’s job and requires no application involvement.

This is the gap that makes keepalive unsuitable as an application-level detector. A database that has entered a livelock, a server whose thread pool is exhausted, and a process stopped under a debugger all acknowledge keepalives indefinitely. The connection is healthy; the service is not. Detecting that requires the application to send something and receive an answer that only a functioning application could produce, which is a heartbeat with a payload and a matching response, as described in heartbeat interval and timeout arithmetic.

The converse is also true and is the reason keepalive remains useful. An application heartbeat cannot distinguish a peer whose process died from a peer whose host vanished from the network, because in both cases no answer arrives. A keepalive failure reported by the kernel carries a little more information: the stack tried and got no acknowledgement, or got a reset, and the error distinguishes those cases. A reset means the peer’s stack is alive and has no state for this connection, which usually means the peer process restarted. A timeout means silence. Neither distinction changes what the application should do, but both are useful for diagnosis, and diagnosis is where the per-observer view of a partition becomes relevant.

Configuring it, and the portability problem

The parameters are exposed differently on each platform, and the differences are large enough to break assumptions.

On Linux the socket option is SO_KEEPALIVE, and the tuning knobs are per-socket options: TCP_KEEPIDLE for the idle time before the first probe, TCP_KEEPINTVL for the interval, and TCP_KEEPCNT for the count. The same names exist on other systems with different availability, and the system-wide defaults live under /proc/sys/net/ipv4/ as tcp_keepalive_time, tcp_keepalive_intvl, and tcp_keepalive_probes. Setting the per-socket options requires enabling SO_KEEPALIVE first; the tuning options do nothing without it.

The portable subset is small. SO_KEEPALIVE is in POSIX and works everywhere; the tuning options are not, and code that sets them needs conditional compilation or a runtime check. A program that wants a specific detection window on an unknown platform cannot rely on the socket options and must either probe at the application layer or accept the platform default.

int on = 1, idle = 30, intvl = 10, cnt = 3;
setsockopt(fd, SOL_SOCKET,  SO_KEEPALIVE, &on,   sizeof on);
setsockopt(fd, IPPROTO_TCP, TCP_KEEPIDLE,  &idle, sizeof idle);   /* Linux */
setsockopt(fd, IPPROTO_TCP, TCP_KEEPINTVL, &intvl, sizeof intvl);
setsockopt(fd, IPPROTO_TCP, TCP_KEEPCNT,   &cnt,  sizeof cnt);

With those values the detection window is 30 seconds of idleness plus three probes ten seconds apart, so roughly a minute. That is a reasonable reaping policy and a poor failover policy, and the difference between those two goals is the whole argument.

There is a further wrinkle on the sending side. A keepalive probe is only sent when the connection is idle in the kernel’s accounting, which includes unacknowledged data. A connection with a persistently full send buffer, because the peer is not reading, is not idle and produces no probes; the application blocks or gets a partial write instead. The failure surfaces as backpressure, not as a keepalive timeout, and the two look nothing alike to the code that has to handle them.

Middleboxes and the probes that never arrive

Keepalive interacts badly with anything in the path that tracks connection state. A NAT device or a stateful firewall keeps a mapping for a connection and expires it after an idle period of its own, which may be shorter than the keepalive idle time. When the mapping expires, the next packet in either direction is dropped or answered with a reset, and the connection dies even though both endpoints are healthy. This is the common reason long-lived idle connections fail in cloud networks, and it is the reason many systems set an application-level heartbeat interval well below the middlebox timeout rather than relying on keepalive at all.

The failure is asymmetric and easy to misdiagnose. The endpoint that sends the first packet after the mapping expired sees a reset or a timeout; the other endpoint sees nothing. From the sender’s perspective the peer looks dead. From the peer’s perspective the connection looks fine until it tries to write. A detector that treats the sender’s error as evidence about the peer will declare a healthy peer dead, and the record of which side observed the failure is the only thing that prevents that conclusion.

The division of labour

Keepalive is the right tool for reclaiming resources held by connections whose peers are gone, and for turning a silent dead connection into an error at the next operation instead of an indefinite hang. It is the wrong tool for deciding that a peer should be replaced, because its timescale is minutes and its evidence is about the kernel rather than the service.

The practical arrangement uses both layers for their respective jobs. Keepalive runs with a window sized to the longest middlebox idle timeout in the path, which reaps genuinely dead connections without generating significant traffic. An application heartbeat runs with a window sized to the failover budget, carrying a payload the peer must process, and its verdict drives membership. The two detect different failures, and the case where they disagree, keepalive healthy and heartbeat failing, is the signature of a process that is alive but not serving, which is the failure mode a connection-level detector structurally cannot see. The pause is the clearest instance, and stop-the-world pauses and false suspicion covers why the kernel keeps acknowledging while the application does nothing.