Distributed Systems · Replication
Replication Topologies and What Each One Tolerates
How single-leader, multi-leader, and leaderless replication differ in where writes are accepted, which node losses they survive, and what conflict handling each one forces.
Replication keeps several copies of the same state on different machines. The copies exist so that losing one machine is not losing the data, and so that reads can be served from somewhere nearby. Neither goal is achieved by copying alone. What a replicated system can survive is decided by a single structural choice: which nodes are allowed to accept a write. That choice partitions the design space into three arrangements, and each arrangement buys availability with a specific, unavoidable bill.
Where a write is allowed to land
In single-leader replication one node accepts writes. It appends each change to a log, assigns it a position, and ships the log to the others, which apply entries in that order. Every write has one total order because there is one writer. Followers are read-only copies that trail the leader by some amount.
In multi-leader replication several nodes accept writes for the same key, usually one per region or data centre. Each leader ships its log to the others, so the system ends up with several independent orders that must be reconciled. Two writes to the same key on different leaders are not ordered with respect to each other, because nothing in the system observed both before either was applied.
In leaderless replication no node is special. A client or a coordinator sends each write to a set of replicas, and the write is considered durable once some number of them have stored it. Reads are sent to a set of replicas and the responses are merged by timestamp or version. There is no failover step because there is no role to fail over.
The three arrangements are distinguished by the answer to “who decides the order”, not by the number of machines. A single-leader group of five and a leaderless group of five have identical hardware and very different failure behaviour.
What the single-leader log buys and costs
The log is the entire mechanism. Because one node assigns positions, a follower that applies entries in order can never see a later write before an earlier one for the same key. Recovery is a matter of comparing log positions: a follower asks for everything after position p and applies it. A newly promoted leader already holds the full prefix, so no reconciliation step is needed.
The cost is that write throughput is bounded by what one node can append and ship. Every write must reach the leader, which means every write pays the network distance to it. In a single data centre that distance is sub-millisecond; between continents it is on the order of a hundred milliseconds, and a synchronous cross-region commit adds roughly one round trip to every write.
The second cost is failover. The leader is a single point of unavailability, and promoting a follower is a decision that requires agreement about which follower is furthest ahead. That decision is the subject of leader leases and fencing tokens, and it is where most of the operational danger in single-leader systems lives.
Which node losses stop progress
Single-leader survives the loss of any follower without affecting writes, as long as at least one follower remains to take over. Losing the leader stops writes until a new one is chosen. Reads served by followers may continue, but they may return data that is behind the lost leader’s uncommitted tail, and that tail may be discarded on promotion.
Multi-leader survives the loss of any leader for writes in the regions that still have one, because each region writes to its own leader. Losing a leader in one region degrades that region to reading from another region, with the corresponding latency. What multi-leader cannot survive cleanly is a partition between leaders, because both sides continue to accept writes and diverge.
Leaderless replication has no failover event at all. Each request names a set of replicas and a threshold; if enough replicas answer, the request succeeds. Losing replicas shrinks the margin. With three replicas, a write quorum of two, and a read quorum of two, the loss of one replica leaves both operations possible but leaves no slack. The loss of two replicas stops writes that require two acknowledgements, even though one healthy copy of the data still exists. This is the characteristic leaderless failure: availability is a function of the count of reachable replicas, not of any node’s role.
| Topology | Write stops when | Reads can be stale because |
|---|---|---|
| Single-leader | leader is unreachable | follower lag |
| Multi-leader | all leaders for the key are unreachable | cross-leader propagation delay |
| Leaderless | fewer than W replicas reachable | fewer than R replicas consulted |
The conflict bill arrives later
Single-leader systems never produce conflicting writes, so they never need conflict resolution. That is a large simplification and the main reason the arrangement remains the default for transactional databases.
Multi-leader and leaderless systems both accept writes that were composed without knowledge of each other. Those writes must eventually be ordered, merged, or one of them discarded. This is the setting in which the term eventual consistency is meaningful: the guarantee is only that, absent new writes, the replicas converge, and the convergence rule is a separate design decision. The cheapest rule is a timestamp per write with last-write-wins, which silently drops the losing write; its correctness depends on clock behaviour and on whether the application can tolerate loss. The more expensive rule records causal history per key so that concurrent writes can be identified as concurrent rather than arbitrarily ordered, which is what version vectors provide.
The bill is not only about correctness. It is about read path complexity. A leaderless read must contact several replicas, compare their versions, and repair the ones that are behind, which is more work per read than asking a single leader. That repair traffic is the subject of Merkle tree anti-entropy when it runs in the background.
The same three nodes under a partition
Consider three replicas holding one key, and a partition that isolates one of them from the other two.
Under single-leader with the leader on the majority side, writes continue and the isolated follower simply falls behind. Under single-leader with the leader isolated, writes stop entirely even though two healthy replicas are reachable. That is the price of a single writer: the system prefers consistency over availability and makes no attempt to hide it.
Under leaderless with a write quorum of two and a read quorum of two, the majority side accepts writes and reads. The isolated replica misses them. When the partition heals, a read that touches the stale replica along with a fresh one will see the older version on one response and the newer on another; the coordinator returns the newer and writes the newer back to the stale replica. If the application had configured W = 1, the majority side would accept a write that the minority replica never sees, and a later read of R = 1 aimed at the minority would return the old value. Quorum arithmetic is what prevents that, and quorum intersection is the exact condition under which it works.
Under two-region multi-leader with the link between regions cut, both regions continue to accept writes to the key. When the link returns, the two versions are concurrent. Nothing in the logs can decide which came first, because no node saw both. The application or the merge function must decide.
Picking by failure model, not by write volume
The decision is usually framed as throughput, but throughput is rarely the binding constraint. The binding constraint is which failures must not interrupt writes and whether the application can express a merge for concurrent values.
If writes must be ordered globally and conflicts cannot be merged, single-leader is the only arrangement that does not push that problem into the application. If writes must continue during a link failure between regions and the data has a natural merge, multi-leader is appropriate, with the understanding that the merge function is now part of the data model. If no single node may be a write bottleneck and the workload is key-value with per-key independence, leaderless spreads the write load and replaces failover with quorum counting.
A hybrid is common and legitimate: single-leader within a region for the ordering guarantees, multi-leader between regions for availability, with the conflict surface narrowed to the keys that genuinely cross regions. The topology can be chosen per key range rather than per cluster, which is why the question “which replication model does this system use” often has no single answer.
References
- Dynamo: Amazon's Highly Available Key-value Store — DeCandia et al., SOSP 2007
- In Search of an Understandable Consensus Algorithm (Raft) — Ongaro & Ousterhout, USENIX ATC 2014
- Eventually Consistent — Werner Vogels, ACM Queue