Distributed Systems · Cluster
Failure Detection
Heartbeats, suspicion, and partial failure.
- 01 Completeness and Accuracy: The Two Properties Every Failure Detector Trades Against Why a detector that never mistakes a live process for a dead one must sometimes wait forever, and how the completeness and accuracy properties make that trade precise.
- 02 Heartbeat Interval and Timeout Arithmetic for Fixed-Threshold Detectors How the choice of heartbeat period and missed-beat threshold sets false-positive rate, detection latency, and message volume at the same time, and why the tail of the delay distribution decides everything.
- 03 SWIM Ping-Request Probes and Why Suspicion Beats Immediate Declaration How a node delegates a probe to a third party before declaring a peer dead, and how incarnation numbers let a wrongly suspected node refute the claim without a restart.
- 04 Phi Accrual Detectors: Turning Inter-Arrival Samples into a Continuous Suspicion Level How a detector fits a distribution to observed heartbeat gaps, converts the current gap into a probability that the peer is dead, and why the exponential form of that probability is what makes a threshold portable.
- 05 TCP Keepalive as a Failure Detector, and Why It Cannot Be One What keepalive probes actually prove about a peer, why the kernel default of two hours makes them useless for failover, and which failures they detect that an application heartbeat cannot.
- 06 Disseminating Suspicion by Gossip: Anti-Entropy, Rumor Mongering, and Merge Order How a suspicion reaches every node without a broadcast tree, why the merge rule must be a total order, and what the log factor in convergence time actually depends on.
- 07 Correlated Timeouts and Detection Storms When Every Monitor Sees the Same Fault Why independent observers are only independent when their failures are uncorrelated, how a shared cause turns many weak suspicions into a confident wrong verdict, and which randomization actually helps.
- 08 Stop-the-World Pauses and the False Suspicion They Cause Why a collector pause, a page-fault storm, or a hypervisor deschedule looks identical to a crash from outside, and how a process can tell the difference using only local evidence.
- 09 Hierarchical Heartbeat Aggregation and the Cost of Intermediate Levels Why a tree of monitors cuts liveness traffic from quadratic to linear, what the extra level adds to detection time, and how the aggregate view loses the identity of the observer.
- 10 Asymmetric Partitions and What a Set of Conflicting Suspicion Reports Means How to read a matrix of who can reach whom, why a single suspicion report cannot distinguish a dead peer from a broken path, and what a system can safely conclude from per-observer evidence.