Heartbeats, Leases, and the Timeout That Declares a Node Dead

Every controller in a cluster acts on the assumption that the objects it reads describe a world that exists. That assumption is only as good as the freshness of the reports behind it, and freshness is established by a timeout, a period after which a silent node is presumed dead. The timeout is unavoidable and it is also, necessarily, wrong in both directions: too short and a slow node is declared dead while it is running; too long and a genuinely dead node keeps its workloads’ identities reserved while nothing serves them.

The mechanism that makes this manageable separates the signal that says “I am alive” from the report that says “here is what I am running”, and it expresses the first as a lease with an explicit expiry rather than as a timestamp that the observer must interpret.

A lease is a write with a deadline

A lease is an object with an owner, a duration, and a renewal time. Its meaning is entirely contained in one comparison: if the current time is more than the duration past the last renewal, the lease has expired and its holder no longer has the right it was granted. Nothing else about the object matters, and nothing needs to be inferred.

The kubelet on each node renews a lease for that node at a fixed interval, which is shorter than the duration by a comfortable margin so that several renewals can be lost before expiry.

apiVersion: coordination.k8s.io/v1
kind: Lease
metadata:
  name: node-1
  namespace: kube-node-lease
spec:
  holderIdentity: node-1
  leaseDurationSeconds: 40
  renewTime: "2026-01-05T10:00:00.000000Z"

The renewal is a single small write to the API server, an update that changes the renewal timestamp and nothing else. It is deliberately small, because its cost is paid once per node per interval, and a cluster with ten thousand nodes at a ten-second interval performs a thousand of these writes per second. Making the renewal carry the node’s full status would multiply that by the size of the status object, which is why the two are separate objects and separate write paths.

The separation is the design’s main concession to scale, and it has a second benefit: the status report and the liveness signal can be produced on different schedules. Status is expensive to compute: it enumerates the node’s conditions, capacity, allocatable resources, and every pod’s status, and is written less often. Liveness is cheap and is written more often. A node whose status write is delayed by a slow API server can still renew its lease, so a busy control plane does not immediately start declaring nodes dead.

The node’s status conditions include a Ready condition, and the node controller sets it from the lease. When the lease has not been renewed within the grace period, the controller marks the node NotReady and applies the corresponding taint. The grace period is the sum of the lease duration and a propagation allowance, and it exists because the controller itself observes the lease through a cache that lags the store.

The three clocks and which one is authoritative

Three time sources are in play: the node’s clock, the API server’s clock, and the controller’s clock. The lease protocol is written so that only one of them decides expiry.

The kubelet stamps the renewal with the API server’s view of time, which it learns from the response headers of its own requests, not from its local clock. A node whose clock has drifted by minutes therefore still renews correctly, because the timestamp it writes is the server’s. The comparison that matters is made by whoever reads the lease against its own clock, and the only reader that makes a decision is the node controller, which runs with the control plane’s clock.

This design choice, writing the observer’s time rather than the subject’s, is what removes clock skew from the failure-detection path. It does not remove it entirely: a kubelet with a badly skewed clock computes its renewal interval against a time source it does not trust, so the interval jitters, and the margin between the renewal interval and the lease duration must absorb that jitter. The margin is typically several multiples of the interval.

The cost of the design is that a node cannot distinguish “my lease is fine” from “my writes are being accepted”. The renewal’s success is the signal; a kubelet that cannot reach the API server has no way to know whether its last renewal was recorded. It proceeds as if it were healthy, which is correct from its own point of view and wrong from the cluster’s. That divergence is the entire subject of split-brain handling.

What happens at expiry, in order

The node controller’s response to an expired lease is a sequence, and the order is what makes recovery possible.

First, the node is marked NotReady and tainted with a NoSchedule effect, so no new workload is placed there. This is reversible and cheap: if the node comes back and renews, the taint is removed and scheduling resumes.

Second, after a further delay, the NoExecute taint is applied. Pods with a bounded toleration for that taint are evicted after their tolerance expires; pods without one are evicted immediately. The delay between the two taints is deliberate: a brief network blip should cost scheduling, not running workloads.

Third, the pods that were on the node are deleted from the API server, but not before the node is confirmed unreachable for long enough that the kubelet cannot be running them. This is the dangerous step, because deleting a pod object while its process is alive creates two copies of the same identity: the old process, still serving, and a new pod scheduled elsewhere. Both may hold the same volume, both may write to the same external system, and if the workload is a singleton with an external lock, the lock is now held twice.

The mechanism that prevents the second copy from being created prematurely is the taint, not the deletion. A pod that has been deleted but whose node is unreachable is still accounted for; the replacement cannot be scheduled onto the unreachable node, and the controller that owns the pod cannot create a replacement until the deletion is observed. The window between “process is alive but unreachable” and “process is gone” is exactly the window in which a false positive causes duplicate work, and its length is the grace period plus the eviction delay.

False positives, false negatives, and why both are permanent

A false positive declares a live node dead. Its cost is duplicated work and, in the worst case, data corruption from two writers. It is bounded by the grace period, because the node’s process is still running and will eventually either reconnect and be re-admitted, or be terminated by its own supervisor.

A false negative leaves a dead node’s workloads accounted for. Its cost is capacity that cannot be scheduled and services that appear to have replicas but do not. It is unbounded in time only if the detection never fires, which cannot happen with a lease-based mechanism as long as the controller is running.

The asymmetry is why the timeout is tuned toward the false-positive side being rare: a longer grace period reduces duplicate work at the cost of longer unavailability. There is no setting that avoids both, and the choice is a statement about which failure the operator prefers. This is a specific instance of the general result that failure detection in an asynchronous system cannot be both complete and accurate: a detector that never suspects a live process must eventually fail to suspect a dead one, and a detector that always eventually suspects a dead process must sometimes suspect a live one.

The practical consequence is that workloads with external side effects need their own fencing. A distributed lock with a lease, a fencing token that a storage system checks, or an idempotent write keyed by a generation number are all ways to make the duplicate-writer window harmless. The orchestrator cannot provide this, because it does not know what the workload’s side effects are; it can only bound the window in which they can happen.

Eviction, disruption budgets, and the node that comes back

The NoExecute eviction is subject to disruption budgets only in the sense that the eviction API respects them; the node controller’s taint-driven eviction does not. A pod whose controller declares a budget permitting no voluntary disruption can still be removed when its node is declared dead, because that removal is not voluntary. The distinction matters when planning maintenance: draining a node respects budgets, and losing a node does not.

When a node returns, the sequence reverses but not symmetrically. The kubelet re-registers, renews its lease, and the taints are removed. The pods that were deleted while it was unreachable are gone from its own view once it reconciles with the API server, the kubelet compares its local pod list against the API server’s and terminates any pod the API server no longer knows about. This reconciliation is the mechanism that prevents a returning node from resurrecting workloads that were rescheduled elsewhere, and it depends on the node being able to reach the API server, which is the same condition whose absence caused the eviction.

A returning node may also find that its pods were replaced on other nodes and that its own copy of a volume is now stale. The orchestrator does not resolve that; the volume’s attach semantics do, and the resolution is that the volume is attached to exactly one node at a time, with the attach operation refusing a second attachment until the first is released. The release is triggered by the pod deletion, which is why the deletion must precede any replacement’s attempt to attach.

The lease as a general primitive

The same object type is used for leader election in the control plane, for coordination between controllers, and for node liveness, with the same semantics and different durations. The uniformity is deliberate: a lease is the smallest object that can express “I hold this until time T”, and every coordination problem in a cluster reduces to holding something until a time that the holder can renew. What differs between uses is only the duration and what happens on expiry: a lost leader stops acting, a lost node’s pods are evicted, a lost scheduler’s assumptions are discarded.

The failure mode that all of these share is the one described above: the holder cannot tell whether its renewal was recorded, so it must assume it was and rely on the observer to enforce expiry. Any component that acts on the assumption that it still holds a lease without checking has reintroduced the split brain the lease was meant to prevent. How the control plane’s own components use this to stay consistent under partition is treated in leader election and control-plane quorum.