Distributed Systems · Consensus
Raft Elections: Terms, Votes, and the Pre-Vote Correction
How terms impose a total order on leadership attempts, why the voted-for rule plus the log-suffix check prevents two leaders in one term, and what pre-vote and check-quorum add.
A Raft cluster has one leader per term, and every log entry carries the term in which it was created. Those two facts, plus one restriction on who may be granted a vote, are enough to rule out two leaders appending entries at the same index. The election mechanism is deliberately mechanical: a follower that hears nothing from a leader for a randomized interval becomes a candidate, increments the term, and asks everyone for a vote. The interest is in why that procedure cannot produce a split brain, and in what the obviously-correct-looking variants get wrong.
Terms as a logical clock
A term is a monotonically increasing integer. Every node stores the largest term it has seen, and two rules govern it: a node that receives a message carrying a term higher than its own adopts that term and reverts to follower; a node that receives a message carrying a lower term rejects it and, if it is a candidate or leader, responds with its own higher term so the sender can step down.
The term is not a wall clock and does not measure time. It partitions the protocol’s history into epochs, each of which has at most one leader. Within a term, the leader is stable; across terms, the leader may change. When a node sees a higher term, everything it believed about the current leader is invalidated, because a higher term means someone started a newer election that the node had not heard about. This is why the term travels on every request and response rather than only on vote requests: an isolated leader that has been superseded will learn about it the moment it tries to contact any node that has moved on.
Terms are also how the log gets a total order that survives leadership changes. An entry is identified by the pair (term, index), and Raft’s Log Matching Property states that if two logs contain an entry with the same index and term, then the entries are identical and all preceding entries are identical. That property is what allows a new leader to repair follower logs by walking backwards, and it is what makes the commit index rule safe.
The two conditions for a vote
A candidate requests votes for its own term. A voter grants the vote only if both conditions hold, and the second one is the subtle part.
The first condition is that the voter has not already voted in this term. Each node records the term in which it last voted and for whom, durably, before responding. The persistence is not optional: a node that grants a vote, crashes, restarts with no memory of the vote, and grants a second vote in the same term to a different candidate has broken the election. With a five-node cluster, two candidates could each collect three votes if votes were not persisted across restarts, and both would become leader in the same term.
The second condition is the log-suffix check. The voter compares the candidate’s last log entry against its own. The candidate’s log must be at least as up to date, defined as: higher last term wins; if the last terms are equal, the longer log wins. A candidate whose log is shorter, or whose last entry is from an older term, is denied a vote even if the voter has nothing against the candidate personally.
The combination yields the Election Safety property. Suppose a leader was elected in term T with a majority. Any later leader in a term greater than T must be elected by a majority, and the two majorities intersect in at least one node. That node voted for the first leader in term T, so by the first condition it cannot have voted for anyone else in term T; it therefore voted in a later term, which means it must have passed the log-suffix check against that later candidate. The proof then needs a lemma: if a log entry is committed in term T, then every leader of every term greater than T has that entry in its log. The lemma follows by induction on terms using the suffix check and the fact that a leader’s log contains everything its electing quorum had. The full argument is in the paper; the operational summary is that the suffix check is what stops a node that missed recent entries from becoming leader and truncating them.
Why the randomized timeout is not cosmetic
If every follower used the same election timeout, every follower would become a candidate at the same instant, every candidate would vote for itself, and no candidate would reach a majority. The cluster would split the vote and immediately start another round with the same alignment. Raft breaks the symmetry by randomizing the timeout over an interval, typically on the order of hundreds of milliseconds in a local network, and re-randomizing on each timeout. The follower that fires first requests votes while the others are still followers, and normally wins before the others time out.
The timeout interval has to satisfy a rough constraint relative to the heartbeat interval: the election timeout must be substantially longer than the heartbeat so that ordinary message jitter does not trigger elections, and short enough that failover is acceptable. The paper’s guidance is a heartbeat an order of magnitude below the election timeout, with the timeout itself randomized across a range of roughly the same order. These are performance settings; getting them wrong costs availability and CPU, not safety.
A candidate that fails to win stays a candidate and retries with a new, higher term after another randomized timeout. There is no separate “candidate timeout” state; the retry is the same transition. This means a partitioned candidate’s term climbs steadily, and when the partition heals its high term forces the legitimate leader to step down and a new election to occur. That is disruptive but safe: the disruption is the reason pre-vote exists.
Pre-vote: stopping the term inflation
Consider a node that has been partitioned away from the cluster for a while. Under the basic protocol it times out repeatedly, increments its term each time, and returns with a term far ahead of everyone else’s. On rejoining it sends a vote request with that term, every node adopts the term, the current leader steps down, and the cluster spends an election cycle recovering from a node that was never going to win. Repeat this for a flapping link and the cluster does little else.
The pre-vote extension adds a preliminary round. Before incrementing its term, a prospective candidate asks the other nodes whether they would vote for it if it incremented. The recipients answer using the log-suffix check but do not change their term or their recorded vote. Only if a majority answers yes does the node actually increment its term and start a real election. A node that has been out of contact cannot pass the pre-vote round, because the current leader’s followers have logs at least as up-to-date, so it never inflates the term and never disrupts the cluster.
Pre-vote changes no safety property; it is purely a liveness and stability improvement. Its cost is one extra round trip before an election starts, which slightly lengthens genuine failover. Implementations that expect long partitions and flapping links enable it, and it is part of the extended Raft description rather than the original paper.
Check-quorum: the leader’s own failure detector
The symmetric problem is a leader that has been partitioned away from its followers. It still believes it is leader, still accepts client requests, and cannot commit anything because it cannot reach a majority. Clients see requests hang rather than fail, and if the leader is serving lease-based reads it may even answer reads from a stale state machine.
The check-quorum mechanism has the leader count responses to its heartbeats over a window equal to an election timeout. If it cannot reach a quorum within that window, it steps down to follower. Stepping down does not restore service, but it converts a hung leader into a node that will accept a new leader’s entries, and it stops the leader from making promises it cannot keep. The window is tied to the election timeout because that is the same interval after which followers would start electing a replacement; aligning the two means the leader gives up at roughly the moment the rest of the cluster moves on.
Neither pre-vote nor check-quorum is in the original protocol description, and neither is required for correctness. Both address the same operational failure: a node whose local view has diverged from the cluster’s, which under the base protocol causes churn or silent unavailability.
Elections interact with everything else
Three interactions are worth stating because they are where implementations diverge.
The first is with log compaction. A candidate’s log-suffix comparison uses the last entry in its log, but a node that has compacted its log no longer has the early entries; it has a snapshot with a last-included index and term. The suffix check must therefore use the snapshot’s boundary when the log is empty, or the candidate will appear to have an empty log and lose elections it should win.
The second is with membership. A vote is counted against the current configuration, and during a joint-consensus transition a candidate must win majorities in both the old and new configurations. A cluster that changes membership while an election is in flight can therefore elect a leader that satisfies the old configuration’s quorum and not the new one, unless the configuration itself is part of the replicated log and takes effect at a defined index.
The third is with the client’s view of leadership. A client that sends a request to a node that has just lost leadership gets a redirect or an error, and must retry. The protocol’s guarantee is that at most one leader exists per term, not that any particular node is the leader, so clients need either a stable leader hint or the ability to follow redirects. A client that retries against the old leader repeatedly will make no progress until it learns the new term.
The election mechanism is small enough to state on one page and subtle enough that the two conditions for a vote are both load-bearing. The durable vote is the one people forget when writing an implementation, because it only matters after a crash, and the log-suffix check is the one people simplify, because it only matters when a lagging node asks. Both failures are invisible in a healthy cluster and both are fatal in a broken one.
References
- In Search of an Understandable Consensus Algorithm — Diego Ongaro, John Ousterhout, USENIX ATC 2014
- Raft Refloated: Do We Have Consensus? — Howard, Schwarzkopf, Madhavapeddy, Crowcroft, ACM SIGOPS OSR 2015
- The Raft Consensus Algorithm — raft.github.io