Snapshot Install and the Cost of Compacting a Consensus Log

A consensus log that is never truncated grows without bound, and a node that has been offline for a week cannot catch up by replaying a week of entries. Both problems are solved by the same mechanism, and the mechanism has a sharp edge: a snapshot replaces history, so a node that receives one loses the ability to answer questions about anything before the snapshot’s boundary. The boundary index is therefore not an implementation detail but the point at which the protocol’s guarantees change from “every entry is available” to “every entry after this point is available”.

Why the log cannot be kept forever

Each entry occupies space on every node that holds it, and the space is multiplied by the replication factor. The state machine, by contrast, typically has a bounded size because it represents a current state rather than a history. A key-value store with a million keys has a state proportional to a million keys regardless of how many writes have been applied; its log is proportional to the number of writes, which is unbounded.

The ratio between the two is what makes compaction worthwhile. A log of a hundred million small writes might be tens of gigabytes, while the state machine they produce is a few hundred megabytes. Replacing the log prefix with a snapshot of the state at the boundary recovers the difference, and the recovered space grows with uptime.

The second motivation is catch-up time. A node that restarts after a long absence must be brought up to date, and replaying the entire log means transferring every entry ever written. If the log has been compacted, the node can instead receive a snapshot of the current state plus the entries after the boundary, which is proportional to the state size rather than to the history size. The difference between replaying a billion entries and installing one snapshot is the difference between a recovery measured in hours and one measured in minutes, though the snapshot itself may be large.

The boundary index and term

A snapshot is described by a last-included index and the term of the entry at that index. The pair plays the role that a log entry plays in the matching procedure: it is the point at which the snapshot’s contents are defined to be equivalent to applying all entries up to and including that index.

The term is required, not decorative. A leader repairing a follower’s log uses the (index, term) pair to find an agreement point. A follower whose log has been compacted has no entry at the boundary index, only the snapshot, so the leader must be able to compare against the snapshot’s term to know whether the follower is at the right point. Without the term, a leader could send entries starting after an index where the follower’s state is actually different, and the divergence would be undetectable until the state machines produced different results.

A node that has just installed a snapshot discards its entire log and sets its commit index to the boundary index. Anything before the boundary is now represented by the state machine’s contents rather than by entries. This is a real loss of information: the node can no longer answer a leader’s request to back up further than the boundary, so the leader must not ask it to.

Install is not AppendEntries

Snapshot installation is a separate RPC, not a large AppendEntries. The difference is structural rather than a matter of message size.

AppendEntries is idempotent at the entry level: a follower that receives an entry it already has ignores it, and a follower that receives an entry conflicting with its log truncates and replaces. The operation is a merge of two logs and it can be retried freely. Snapshot installation is destructive: the follower replaces its entire state machine and log. A retried install that arrives out of order with another install can leave the follower with a snapshot older than one it already had, and the protocol must reject it.

The rule is that a follower accepts a snapshot only if its last-included index is greater than the follower’s current commit index. A snapshot at or below the commit index is discarded, because the follower already has that state and possibly more. A snapshot above the commit index replaces the follower’s state entirely, including any log entries it holds, because those entries may conflict with the snapshot’s contents.

The destructive nature also means the install cannot be partially applied and then abandoned. A follower that receives half a snapshot and then loses its connection has a state machine in an undefined condition unless the implementation stages the snapshot outside the live state and swaps it in atomically. The common approach is to write the snapshot to a temporary location, verify it, and rename it into place, so that a crash mid-install leaves the old state intact.

Chunking and the transfer cost

Snapshots are large, and sending one in a single message is impractical both because of message size limits and because a single lost packet would require retransmitting the whole thing. Implementations chunk the snapshot into fixed-size pieces, typically on the order of hundreds of kilobytes to a few megabytes per message, and send them in sequence with an offset.

Chunking introduces a liveness interaction. A follower that is receiving a snapshot is not participating in the normal replication stream, so its acknowledgements do not advance the leader’s commit index. If the snapshot transfer is slow, as when a large state crosses a slow link, the leader may be unable to commit new entries during the transfer, because the follower’s vote is needed for the quorum. The usual mitigation is to have the leader continue sending entries to other followers while the snapshot goes to the lagging one, so that the quorum is maintained without the lagging node.

There is a subtler interaction with leadership changes. A snapshot transfer that spans an election is interrupted, because the new leader has a different view of the follower’s state and may not have the snapshot the old leader was sending. The transfer must restart under the new leader, and the follower must discard the partial snapshot. This is why snapshot installs are expensive to interrupt and why a cluster with frequent elections and large states can spend most of its time re-transferring snapshots.

When to snapshot

The trigger is usually a log size threshold rather than an entry count, because the cost of replaying a log is proportional to its bytes. A threshold of some tens of thousands of entries is common in the published descriptions, but the right value depends on entry size and on how much replay time is acceptable at startup.

Snapshotting is not free even when it is not transferring. Producing a consistent snapshot of a live state machine requires either stopping the apply loop while the state is copied, or a copy-on-write mechanism that lets the snapshot be taken while writes continue. The first costs a latency spike proportional to the state size; the second costs memory for the copied pages. Neither is free, and a cluster that snapshots too frequently pays the cost repeatedly while gaining little, since the space saved is the difference between the log since the last snapshot and the state size.

The compromise most implementations reach is to snapshot when the log exceeds a multiple of the last snapshot’s size, so that the space reclaimed is at least comparable to the space the snapshot occupies. That ratio keeps the total space bounded at roughly twice the state size plus the recent log, rather than growing with the snapshot frequency.

Snapshots and the configuration

A snapshot must carry the cluster configuration in effect at its last-included index. A node that joins by installing a snapshot has no configuration entries in its log, so the snapshot’s metadata is its only source of truth about which quorum to count against. An implementation that stores the configuration separately from the snapshot can install a snapshot whose boundary predates a configuration change and end up with a node that believes it is in a configuration the cluster has left.

The same applies to the membership change machinery. A joint-consensus transition that is in progress when a snapshot is taken must be represented in the snapshot, either by including the joint configuration as the snapshot’s configuration or by refusing to snapshot during a transition. Refusing is simpler and is what several implementations do, at the cost of delaying compaction until the transition completes.

What compaction does to the guarantees

Before compaction, a node can answer any question about the log: it has every entry, so it can reconstruct any past state and can participate in any log-repair conversation. After compaction, the node’s knowledge begins at the boundary. It cannot serve a follower that is further behind than the boundary except by sending a snapshot, and it cannot answer a query about a value that was overwritten before the boundary.

This is a genuine reduction in the protocol’s capabilities, and it is accepted because the alternative is unbounded growth. The practical consequence is that a cluster’s ability to recover a very stale node degrades over time: the node can be recovered, but only by a full state transfer rather than by incremental catch-up. For a node that has been offline long enough, the state transfer is the same operation as provisioning a new node, which is why the operational procedure for a long-absent node is usually to wipe it and add it as a new member rather than to let it catch up.

The other consequence is that the commit index and the snapshot boundary must be reconciled carefully at startup. A node that restarts with a snapshot and a log containing entries after the boundary must set its commit index to at least the boundary, because the snapshot’s contents are by definition committed. An implementation that starts with a commit index of zero and waits for the leader to advance it will serve reads from a state machine it believes is empty while the snapshot says otherwise.

Snapshot as the unit of transfer

The last property worth noting is that the snapshot is the only mechanism by which a node can receive state it never had. Log entries describe transitions; a snapshot describes a state. A node that has lost its log entirely, or that has joined the cluster for the first time, needs the state, and no sequence of entries can supply it without replaying the entire history.

This is why snapshot transfer is on the critical path for adding capacity. A cluster that wants to add a node must transfer the full state to it, and the transfer time is bounded by the state size and the link bandwidth rather than by the protocol’s message rate. The quorum arithmetic says the new node will eventually count toward majorities; the snapshot mechanism determines how long “eventually” takes.