Leader Election, Leases, and What Quorum Protects Against

Several controllers in a cluster must run in exactly one place at a time, and the requirement is not about throughput. A controller that creates children for an owner is safe to run twice only if it is idempotent, and the ownership mechanism does not enforce idempotence, and two instances that each compute a desired set and each create it will produce two sets. The scheduler is a sharper case: two instances binding the same workload race on a conditional write, and the loser’s work is wasted, but two instances that each maintain their own assumed-state cache make decisions against different pictures of the cluster. Single-active execution is therefore a correctness requirement for a subset of components, and the mechanism that provides it is an election over a lease.

The interesting part is not the election. It is the bound on how long two instances can both believe they are active, and what happens when the store that arbitrates is itself partitioned.

The lease-as-lock, and why it is not a mutex

A leader election built on a lease works like this. Each candidate attempts to create or acquire an object with a holder identity, a duration, and a renewal time. The one that succeeds holds the lease until it either renews past the duration or releases it. Others poll, and one of them acquires it when the holder’s lease has expired.

The mechanism is a lock with a timeout, and its semantics differ from a mutex in one crucial respect: the holder cannot be prevented from acting after its lease has expired, because the holder’s own clock and the store’s view of time are not the same. A holder that has been partitioned from the store continues to believe it holds the lease, and it will continue to act. The new holder, having observed the expiry, also acts. The overlap is real and unavoidable, and the design’s job is to bound it rather than to eliminate it.

Two things bound it. First, the holder checks the elapsed time since its last successful renewal against the lease duration, using a clock it trusts locally, and stops acting when that interval is exceeded. The check must happen before every action, not once per loop, because a loop iteration can be long. Second, the duration is set well above the renewal interval, commonly a small multiple, so that a single lost renewal does not cause a spurious step-down, and the holder’s local timer expires only after it has genuinely failed to renew for the full duration.

The residual overlap is the interval between the holder’s last successful renewal and its local timer expiring, which is at most the lease duration, plus the propagation delay for the new holder to observe the expiry. So the overlap is bounded by roughly one lease duration, and that is the number to reason about: what can the old leader do in that window, and is it harmful?

For a controller creating children, the answer is usually benign: a duplicate child is an extra object that the next reconcile deletes. For a controller that allocates a scarce resource or performs a non-idempotent external action, the window must be covered by fencing at the resource, not by the election.

What the election actually reads and writes

The election’s correctness depends entirely on the store’s consistency model, and this is where implementations diverge.

If the store is a single API server backed by a consensus-replicated key-value store, then a read of the lease is linearisable with respect to writes: the value returned is the latest committed value, and two clients cannot observe different holders. This is what makes the acquire operation safe. An acquisition is a conditional write, meaning “set the holder to me only if the current holder is empty or the lease has expired”, and its evaluation happens at the store, against the committed state, not at the client. Two candidates racing both send the conditional write; exactly one succeeds, and the loser receives a conflict. No lock manager is needed because the compare-and-swap is the arbitration.

If the store is merely replicated without consensus, or if the client reads from a stale follower, the election degrades in a specific way: a candidate can observe an expired lease that has in fact been renewed, and acquire it. Two holders then exist with no overlap bound at all. This is why the requirement is stated on the store rather than on the election code, and why systems that use a consensus store for coordination, the design ZooKeeper made explicit, treat the store’s sequential consistency guarantee as the primitive and build everything else from it.

ZooKeeper’s contribution was to expose a small set of ordered primitives (sequential nodes, watches, and a total order on writes) and to show that leader election, group membership, and configuration management all reduce to them. The pattern it popularised for election is worth stating because it avoids a subtle problem: rather than racing on a single lock node, each candidate creates an ephemeral sequential node and the candidate with the lowest sequence number is the leader. The ordering removes the race, and the ephemeral property ties the node’s existence to the session, so a crashed candidate’s node disappears without any timeout being consulted. The timeout is replaced by session expiry, which is the same idea with the same false-positive cost.

Raft-based stores take the other route: they make the store itself the arbiter and expose linearisable compare-and-swap, so the election is a handful of conditional writes. The trade is that the store now requires a majority of its members to be available for any write, including a lease renewal. A control plane with a three-member store loses write availability when two members are unreachable, and a leader whose renewal fails steps down even though it is healthy, because the failure is in the store, not in the leader.

Quorum, and the failure it actually prevents

Quorum is often described as protecting against split brain, and that description is imprecise in a way that matters. A majority requirement does not prevent two leaders; it prevents two committed leaders, by making it impossible for two disjoint sets of members to each commit a write. Any write that commits has been acknowledged by a majority, and two majorities of the same set intersect in at least one member, so the second write cannot be committed without the intersecting member having seen the first.

What this guarantees is that the store has a single history. It does not guarantee that a client’s actions are serialised with that history. A leader that committed a write and then lost contact with the store still has the effects of that write in the outside world, and a new leader that never saw the write is free to act on a different view. Quorum protects the log; the outside world needs its own protection.

The availability cost of quorum is direct: a cluster of 2f+1 members tolerates f failures and needs f+1 members to make progress. Three members tolerate one failure; five tolerate two. Adding a member increases read capacity and decreases the failure probability of the majority, but every additional member also adds a round trip to the commit path if the leader waits for all acknowledgements, which is why the commit rule counts a majority rather than all. A five-member store is not twice as available as a three-member one; it is more available under independent failures and slower on every write.

The placement of the members matters as much as the count. A three-member store spread across three availability zones tolerates the loss of one zone; the same three members in one zone tolerate nothing that affects the zone. And a member that is reachable but slow is worse than one that is down, because it participates in the majority while adding latency to every commit. The standard remedy is to require the leader to be within an election timeout of the followers and to step down otherwise, which converts a slow leader into a new election.

The election loop’s practical shape

An implementation has three states and two timers, and the timers are the whole design.

A candidate that has no lease attempts acquisition, backing off between attempts. The backoff matters: without it, a set of candidates that all lost an election retry in lockstep and produce a burst of conflicts on the store at every expiry.

A leader renews at a fixed interval well below the duration, and each renewal is a conditional write that fails if the holder field no longer names it. A failed renewal is not immediately fatal, since the leader retries within the remaining margin, but when the local deadline passes, it stops acting and transitions to a follower state. The transition must be observable to the rest of the process: a controller that continues its current reconcile after the lease has been lost has acted outside the lease, and the reconcile’s own duration must therefore be bounded or made checkable.

A follower polls, and its polling interval determines how long the cluster is without a leader after an expiry. Too long and the control plane is idle for a period after every failover; too short and every follower is a steady load on the store. The usual compromise is a poll interval a fraction of the lease duration, with jitter to avoid synchronised polling.

The subtle failure is a leader that is alive and renewing but cannot do its work, blocked on a downstream call, or starved of CPU. It holds the lease, so no other instance takes over, and the component is effectively down while appearing healthy. Nothing in the election detects this, because holding a lease proves only the ability to write to the store. The mechanism that detects it is a separate health signal, and the decision to release a lease because the holder is unhealthy is a policy choice that trades availability for progress.

Where election is the wrong mechanism

Election is for components that must be singular. It is the wrong tool for components that can be sharded, because a sharded controller gets its safety from partitioning rather than from exclusion: the instance responsible for a hash range is the only one that touches those objects, and no lease is needed as long as the partition is stable. The partition can be maintained by the same consensus store as a lease per shard, which gives finer failover granularity at the cost of more objects and more renewal traffic.

It is also the wrong tool for anything whose correctness depends on acting at most once. The overlap window is bounded but not zero, and a component with a non-idempotent external effect must fence that effect independently: a generation number checked by the resource, an idempotency key, a lease on the resource itself. Election reduces the frequency of double execution; it does not make double execution safe.

What election does provide is a single writer for the objects whose deletion and creation must be ordered, and that is the same property the ownership mechanism relies on when it assumes only one controller creates children for a given owner. It is the connection drawn in controller ownership and the garbage collector. The lease it uses is the same object described in heartbeats, leases, and node failure detection, with a longer duration and a different consequence on expiry: a node’s lease expiring evicts its workloads, while a controller’s lease expiring merely stops it from acting.