Replicating the Commit Decision So the Coordinator Can Die

Two-phase commit’s blocking behaviour traces to one property: the decision exists in exactly one place. While that place is unreachable, participants that voted yes cannot act. The obvious repair is to put the decision in more than one place, and the only way to do that safely is to have the decision be the outcome of a protocol whose result survives the loss of any minority of its members. That is consensus, and using it for commit changes both the failure model and the cost.

The decision as a log entry

Reframe the coordinator’s role. Instead of a process that collects votes and writes a record, the coordinator is a client of a replicated log. The log entry it wants to append says “transaction X commits”. A majority of the log’s replicas must store the entry before it counts as decided, and once a majority has it, the entry is durable against the failure of any minority and readable by any future leader.

The participants’ votes become part of the entry’s content. The leader collects votes, and only proposes a commit entry if every vote was yes; a no vote produces an abort entry or, under presumed abort, no entry at all. The proposal is then replicated, and when the leader learns that a majority has accepted it, it sends commit to the participants.

This is the shape of Paxos Commit: the commit decision is a value chosen by an instance of the Paxos algorithm, and the participants’ votes are the value being agreed on. The participants still vote and still hold their prepared state; what changes is that the decision is no longer held by one process. The single-record version of the protocol, and why its decision log is the whole of its failure surface, is described in two-phase commit and the coordinator decision record.

The recovery property that follows is the important one. A new coordinator can be elected from the log’s replicas, read the log, and find the decision for every transaction that reached one. Participants that are still prepared ask the new coordinator and get an answer. The window in which no answer exists shrinks from “until the old coordinator’s process restarts and its disk works” to “until a majority of the log’s replicas are reachable”. The latter is a condition the system can monitor and, in a well-designed deployment, the normal state.

What the extra replication costs

The cost is a second distributed protocol nested inside the first. Instead of one durable write at the coordinator, the decision requires a round trip to a majority of the log replicas and a durable write at each of them. If the log is replicated three ways, a commit decision costs two durable writes at remote replicas before the coordinator can tell anyone.

That cost is on the critical path of every transaction, which distinguishes it from a background replication scheme. The comparison with two-phase commit alone is: one local durable write and two message round trips, versus a majority round trip for the decision plus the two message round trips. In a local network the extra majority round trip may be a fraction of a millisecond; across a wide area it is the full latency of the farthest replica, and the choice of where to place the log replicas becomes a latency decision.

The usual engineering resolution is to place the log’s replicas near the coordinator and accept that the log’s availability domain is smaller than the data’s. A transaction whose participants span three regions can still have its decision recorded in a majority of replicas in one region, which keeps the decision’s latency local at the cost of tolerating only the failure of that region’s minority. If the whole region fails, the decision is unavailable even though the participants are alive, and the transaction is in doubt again. That is a strictly better failure domain than a single process, but it is not the same as surviving the loss of any region.

There is a further subtlety in how the log’s entries interact with the participants’ prepared state. A commit entry in the log is a decision, but a participant learns it only when the coordinator sends the message or when the participant reads the log. If participants are allowed to read the log directly, the coordinator’s message becomes an optimisation rather than a requirement, which reduces the failure surface: a participant that can reach any log replica can resolve itself. That is the design that makes in-doubt transactions rare rather than merely recoverable, and it requires the log to be readable by the participants, which means the log’s read path must be available at least as widely as the participants.

Leader changes during a decision

Consensus protocols handle leader failure by electing a new leader, and the election has an interaction with commit that does not appear in single-node two-phase commit.

Suppose the coordinator leader has collected all votes and is about to append the commit entry. It crashes before appending. A new leader is elected. It reads the log, finds no commit entry for the transaction, and must decide what to do. The participants have voted yes and are holding locks; from the new leader’s perspective, the transaction is simply undecided. It can propose an abort, and the participants will accept it. This is correct: no commit entry existed, so no participant could have committed, and abort is consistent.

Now suppose the leader crashed after appending the entry but before the entry was replicated to a majority. The entry is on the old leader’s disk and possibly on one follower’s. The new leader’s log may not contain it, and by the consensus protocol’s rules, the new leader’s log is authoritative: entries not on a majority are discarded. The transaction aborts even though one node has a commit entry on disk. This is why the entry must be replicated to a majority before the coordinator tells anyone to commit: the message must never precede the durability of the decision. A coordinator that sends commit after a local append but before majority replication creates exactly the inconsistency that consensus is meant to prevent.

The third case is the one consensus gets right and single-node two-phase commit gets wrong. The entry was replicated to a majority, the leader crashed before sending the commit messages, and a new leader is elected. The new leader reads the log, finds the entry, and resends. The participants commit. The transaction survives the total loss of the old leader, including its disk, because a majority of the log has the entry.

The vote’s fate in the log

A design question that separates implementations is whether the votes themselves go into the log. If only the decision is logged, the new leader cannot distinguish “all votes were yes and the decision is commit” from “some votes were no and the decision is abort” except by reading the decision. That is enough, because the decision is what matters. But it means the new leader cannot finish a transaction that was interrupted before any decision was recorded, because it does not know the votes.

If the votes go into the log, each vote as an entry, the new leader can read the votes, determine whether all were yes, and complete the protocol. This is the Paxos Commit structure, where each participant’s vote is a value that the log agrees on, and the decision is a function of the agreed votes. It makes recovery complete: any transaction that reached the vote phase can be finished by a new leader without consulting the participants. The cost is more log entries per transaction, which is proportional to the number of participants.

The trade is between log bandwidth and recovery completeness. Logging votes means every transaction writes n+1 entries (one per vote, one decision) rather than one. For a coordinator with many participants, that is a large multiplier on the log’s write load, and the log becomes the throughput bottleneck of the whole system. Systems that choose this path usually reduce the participant count instead, which brings the design back to the partitioning argument: fewer participants per transaction is worth more than any logging optimisation.

The composition with the participants’ protocol

Replicating the decision does not change the participants’ half of two-phase commit. They still prepare, still vote, still hold locks, and still wait for a decision. What changes is who they can ask. A participant that cannot reach the coordinator can ask any log replica, and if the decision was recorded, it gets an answer. The prepared state’s duration is now bounded by the log’s availability rather than by a process’s.

That has an operational consequence worth stating. The set of nodes whose failure causes blocking is now the log’s majority, not the coordinator. Blocking is not eliminated, only relocated, and the participants’ side of it is unchanged: see in-doubt transactions and blocking. Capacity planning for the log’s replicas becomes as important as planning for the data nodes, and a log replica that is slow to flush lengthens every transaction’s commit. The log is small and hot, which is a different resource profile from the data, and treating it as just another service is a common way to build a system whose commit latency degrades under load.

The other consequence is that the participants’ recovery queries become read traffic on the log. A burst of coordinator failure produces a burst of queries from every prepared participant, all asking about transactions that may or may not have decisions. The log must handle that read burst without delaying the writes that are still in flight, which is a read-write contention problem inside the consensus implementation rather than in the transaction protocol. Systems that separate the log’s read path from its write path, or that let participants learn decisions through a streaming interface rather than a query, avoid the spike.

When the complexity is justified

The justification is narrow and specific: the system must keep making progress when the coordinator’s process and disk are both lost, without human intervention. If the coordinator is a process on a machine whose disk is mirrored, and the operational procedure for a lost coordinator is to restore from a backup and replay, then replicating the decision buys recovery time rather than correctness, and the extra round trip on every commit may not be worth it.

If the coordinator loss is a routine event, whether because it runs on commodity hardware, because the deployment is large enough that some coordinator is always failing, or because the failure domain spans a region, then a single durable record is not a decision site at all, it is a bet. Replicating the decision converts that bet into a protocol, and the cost is one majority round trip per commit, paid whether or not anything fails.

The middle ground that many systems take is to replicate the decision but keep the vote collection unreplicated. The votes are transient, the decision is durable, and the recovery rule is that an undecided transaction aborts. That is presumed abort with a replicated decision, and it gives most of the availability benefit for a fraction of the log volume. The residual gap is the transaction that had all yes votes and crashed before the decision was recorded: it aborts, and the work is lost, but no participant is left holding locks indefinitely.