Quorum Placement and the Latency of the Second-Fastest Replica

Consensus latency is not set by the number of replicas. It is set by the placement of the replicas relative to the client and to each other, and specifically by which subset of them forms the quorum that must respond. A three-node group with one replica on another continent can have the same commit latency as a three-node group entirely inside one datacenter, or it can be several times slower, depending on which two nodes are near the client and whether the leader is one of them. The arithmetic is simple enough to do before deploying, and getting it wrong is expensive to fix later because it is a data-placement decision.

Majority quorums and the second responder

In Raft and in Multi-Paxos, a write is committed when a majority of the voting members have durably stored it. For a five-node group, that is three; for three nodes, two. The leader sends the entry to all followers in parallel and waits for enough acknowledgments. The latency of the commit is therefore the time until the fastest majority responds, which for n nodes and quorum size q = floor(n/2) + 1 is the q-th smallest response time.

For three nodes and a quorum of two, the commit latency is the second-fastest response. That single fact drives everything else. The slowest replica is irrelevant to the commit as long as the two fastest respond, which means one distant replica can be tolerated at no latency cost — provided the leader and at least one other fast replica are near each other.

Consider three placements with the same three nodes:

Placement	Replica locations	Second-fastest RTT (approx.)
All local	one datacenter, three AZs	1–2 ms
Two near, one distant	two in region A, one in region B	1–2 ms
Spread evenly	one in each of three regions	inter-region RTT, 30–80 ms

The second row is the interesting one. A replica in another region is present for durability — it can form a quorum with either local node if one fails — and it does not affect commit latency in steady state. The third row pays the cross-region floor on every commit, because the second-fastest of three equidistant nodes is one round trip away.

This is why “three regions for durability” and “three regions for the write path” are different designs. The first is achievable with two near replicas; the second is not.

What “near” has to mean

The near replicas must be near each other and near the leader, because the leader is the one collecting acknowledgments. If the leader is elected in the distant region, the two local replicas are now each one cross-region hop from it, and the commit latency becomes the cross-region RTT even though the data is mostly local.

Leader placement is therefore part of the placement decision. Raft elects whichever candidate gathers a majority of votes first, and vote latency follows the same geometry: a candidate in a region containing two of three nodes can win with local votes, while a candidate alone in a distant region needs a cross-region vote. Pre-vote and leader-stickiness extensions reduce churn, but they do not change the underlying preference for a leader in the majority-dense region. In a two-near-one-far layout, the leader will normally be local, and the design is stable. In an even spread, leadership can land anywhere and the latency is uniformly bad.

Reading and the leader lease

Reads have their own geometry, and it is often the larger share of traffic. A linearizable read in Raft requires either a round trip through the log (append a no-op, wait for a majority) or a leader lease, which lets the leader answer locally on the strength of a time-bounded guarantee that no other leader has been elected.

The lease converts read latency from a quorum round trip to a local operation, at the cost of a correctness assumption: the lease must expire on the leader before any new leader can be elected, which requires bounded clock drift and bounded message delay. The lease duration must therefore exceed the maximum election timeout plus the maximum one-way delay, with margin. Setting it too short produces stale reads during a partition; setting it too long extends the window after a leader failure during which reads are refused.

A follower read is a third option and is the one that matters for multi-region read scaling. A follower can serve a read that is allowed to be slightly stale — bounded by how far behind it is — without contacting the leader at all. This is what makes read-heavy multi-region designs work: writes pay the quorum cost, reads are served locally with a staleness bound the application accepts. The bound is measured as log position, not as time, for the reasons covered in replication lag as a recovery point bound.

The client’s own distance

The commit latency above measures the leader’s round trip to its followers. The client’s latency is that plus the client’s round trip to the leader. A client in a third region pays two cross-region legs: one to reach the leader, one for the leader’s quorum round.

client RTT to leader          40 ms
leader -> quorum round trip   40 ms
total write latency          ~80 ms  (serialized, not overlapped)

The two legs are sequential because the leader cannot begin collecting acknowledgments before it has the request. This is why a design that places the leader “centrally” for fairness is usually worse than one that places it where most writes originate: the sum of two half-floors is larger than one floor plus a small asymmetry for the minority of clients.

A useful rule for write-heavy workloads is to place the quorum in the region with the majority of writes and accept that distant clients pay two legs. A rule for read-heavy workloads is to place a follower near every large client population and serve reads locally. Trying to optimize both simultaneously with one placement is where designs become incoherent.

Flexible quorums and the read/write split

Standard majority quorums require every read quorum and every write quorum to intersect. Flexible Paxos observes that the requirement can be relaxed: it is sufficient for each read quorum to intersect each write quorum, not for write quorums to intersect each other. If writes require a quorum of size W and reads require size R from a group of n, intersection holds when W + R > n.

For a five-node group with one replica in each of five regions, a write quorum of four and a read quorum of two satisfies 4 + 2 > 5. Writes become more expensive — four of five responders, so the fourth-fastest latency — and reads become cheaper. The reverse split, W = 2, R = 4, makes writes cheap and reads expensive. The trade is explicit and it is the right knob when the workload is asymmetric.

The catch is that the relaxed write quorum is a durability decision as much as a latency one. A write quorum of two out of five tolerates three simultaneous failures for availability, but the data is on only two nodes, and a subsequent failure pattern that loses both loses the write even though the cluster is still running. The intersection rule protects consistency, not durability against correlated failure. Placement within a failure domain — one replica per availability zone, never two on the same rack — is what protects durability, and it is independent of quorum size.

Witnesses and non-voting members

A common way to get cross-region durability without cross-region latency is to add a witness: a member that stores the log but does not vote, or votes only in a tie. It can break a tie in a two-node group without adding a third full replica’s cost, and it does not sit in the fast path because the two full members can form a majority on their own.

The failure mode is that a witness which becomes a voter during a reconfiguration changes the latency profile silently. Reconfiguration is the most delicate part of consensus: a membership change from two voters plus a witness to three voters spread across regions turns a 2 ms commit into an 80 ms commit, and the change is applied by the same log that is now slower. Any automated reconfiguration that moves voters between regions must be treated as a latency-affecting change, not as a capacity operation.

Placement as a constraint, not an optimization

The practical conclusion is that replica placement should be decided from the failure domains and the write origin distribution, and then checked against the quorum arithmetic, rather than chosen to minimize some average.

A defensible procedure:

1. Enumerate the failure domains that must be survivable: rack, availability zone, region, and any shared dependency such as a single network provider or power feed.
2. Choose a replica count that lets a quorum form with the largest tolerable number of domain failures.
3. Place replicas so that at least q of them are in the lowest-latency domain containing the write majority, and at least one is outside it.
4. Compute the second-fastest RTT for the resulting layout, and confirm it against the write latency budget.
5. Confirm that leader election cannot produce a leader outside the fast domain for longer than the budget allows.

Step 5 is the one most often omitted, and it produces the confusing symptom of a system that is fast most of the time and slow after every failover. The placement was right; the election outcome was not constrained. Pinning leadership to a region, or weighting votes so that a candidate outside the fast domain cannot assemble a majority without it, closes that gap at the cost of making the fast domain’s availability a prerequisite for writes — which is exactly the trade that the failover time budget has to account for.