Stop-the-World Pauses and the False Suspicion They Cause

From the outside, a process in a long garbage collection pause and a process that has been killed differ in no observable way. Both stop sending heartbeats. Both stop accepting connections. Both leave their sockets open, because the kernel keeps them open even when no thread is running to service them. A detector watching the network cannot tell them apart, and any detector that acts on the observation will occasionally act on a live process.

The asymmetry is that the process itself knows. During a pause it is not running, so it cannot report anything at the time, but the pause leaves a trace it can read afterwards: the wall-clock time when it resumes minus the last time it recorded running. That local measurement is the only evidence that distinguishes a pause from a death, and the design question is how to get it to the observers before they act.

What the pause does to the socket

The kernel’s TCP stack is not stopped by the application’s pause. Incoming segments are acknowledged by the kernel and placed in the receive buffer, up to the window size, whether or not the application ever reads them. A keepalive probe is answered by the kernel, which is why keepalive cannot detect a paused process. An application heartbeat, by contrast, requires the application to construct and send a message, and a paused application sends nothing.

On the sending side the pause produces a different signature. Messages already written to the socket leave, because the kernel transmits them. Messages the application would have written during the pause do not exist. When the application resumes, it typically has a backlog of timer events and scheduled work, and it processes them in a burst. An observer watching the stream sees a gap and then a burst of catch-up messages, which is a recognizable shape but arrives too late to prevent the suspicion that the gap already triggered.

The burst also creates a second-order effect. A paused node that is a member of a cluster resumes and sends a large batch of stale heartbeats or gossip updates, and the observers that had already marked it dead receive a flood from a node they consider gone. Whether they accept the flood depends on the identity and incarnation rules, which is the interaction described in suspicion and incarnation refutation.

Sizing the timeout against pause length

The naive response is to set the timeout above the longest pause. That works and it is expensive, because the timeout that covers the worst pause also delays every genuine detection by the same amount. If the worst pause is ten seconds and the target failover time is two, the requirement cannot be met by a timeout alone.

The reason the worst pause is long is usually structural. A collector whose heap is large and whose live set is nearly the whole heap has a pause proportional to the live set, and the live set is a function of the workload, not of the collector’s tuning. A full collection on a heap of many gigabytes can pause for a substantial fraction of a second and, in unfavourable cases, much longer. A virtual machine that is descheduled because the host is oversubscribed can pause for an unbounded time, because the host’s scheduler is not part of the guest’s model. Neither of these is a bug in the detector, and neither is fixable by tuning the detector.

The structural fixes are on the paused process’s side. A collector with concurrent marking or region-based evacuation bounds the pause by design rather than by luck. A runtime that can report its own pause history can tell an observer what happened. A process that must not be descheduled needs a host that does not oversubscribe, which is a deployment property rather than a protocol property.

Local health as a second signal

The observer’s problem is that it has one signal, the absence of heartbeats, and one hypothesis, death. The paused process has a second signal available: its own recent scheduling history. If it can communicate that history, the observer gets a way to distinguish the hypotheses.

The mechanism is to have the process measure its own responsiveness. A background thread, or a timer that runs at a high priority, records the intervals between its own executions. When the process resumes after a pause, it knows the pause duration from the gap in those records, and it can attach that knowledge to the next heartbeat it sends. The observer, on receiving a heartbeat that says “I was unresponsive for 4.2 seconds ending 200 milliseconds ago”, can decide that the gap was a pause rather than a death and can suppress the suspicion.

This is the Lifeguard approach: local health awareness, where the monitored process reports its own view of its health alongside the heartbeat, and the detector uses that report to modulate its suspicion. The design constraint is trust: the report is self-reported, so a process that is lying, or that is so broken it misreports, can suppress its own detection. That is acceptable for pauses, which are the dominant false-positive cause, and unacceptable for a process that has actually failed, because a failed process sends nothing at all. The signal only matters in the case where a heartbeat arrives after a gap, and in that case the process is demonstrably running.

The subtlety is that the report must be about the recent past, not the present. A process that is healthy now may have been paused for the previous five seconds, and the observer needs to know about the five seconds, not about the current instant. The heartbeat therefore carries a summary of the interval since the last heartbeat, and the observer uses that summary to decide whether the gap it observed was the pause the process is describing or something else.

Distinguishing pause from partial failure

A pause is total: the process does nothing. A partial failure, where some threads are blocked and others run, is different, and it defeats the local-health approach because the health thread keeps running and reports health while the service threads are stuck.

The signature of a partial failure is that heartbeats continue while the work stops. A detector that only watches heartbeats sees a healthy node. A detector that watches a heartbeat with a payload the peer must process, or that tracks the completion of real work, sees the difference. This is why the useful heartbeat carries something that requires the service’s own machinery, not just the runtime’s scheduler.

The same distinction applies to a process that is running but whose dependencies are gone. It sends heartbeats on time and cannot serve a request, because the database it talks to is unreachable. From the detector’s perspective the node is healthy; from the service’s perspective it is useless. Detecting that requires either a dependency check inside the heartbeat handler, which couples the heartbeat to the dependency and makes the heartbeat’s failure ambiguous, or a separate health signal that reports dependency state as a distinct field. Conflating the two produces the worst outcome: a heartbeat that fails because a dependency is slow, and a detector that declares the node dead for a reason that has nothing to do with the node.

Where the pause still wins

No amount of local reporting fixes the case where the pause outlasts the observer’s patience. If the observer declares the node dead after five seconds and the node resumes after thirty, the declaration stands, and the local report arrives attached to a heartbeat the observer may not accept. The system must therefore have a recovery path that is cheaper than the original declaration, which is the argument for quarantine over removal in correlated timeouts and detection storms.

There is also a case where the pause is not a pause but a swap storm, a page-fault loop, or a storage stall inside the process. These look like pauses from the outside and are not reported as pauses by the runtime’s own timing, because the runtime’s timer thread may also be blocked on the same resource. The process’s self-report is only as good as the independence of the thread that produces it, and a thread that shares the stalled resource reports the stall as a pause of the whole process, which is at least honest.

The final limitation is that all of this is per-observer. A node that pauses for ten seconds is suspected by every observer that has a five-second threshold, and the fact that each suspicion was individually reasonable does not make the aggregate verdict correct. The observers need a way to compare notes, and comparing notes is only useful when their observations are independent, which a shared pause is not. That is the correlated-failure problem, and the pause is its most common instance.