Bootstrapping a Replica from a Snapshot and a Log Tail

A replica that has been offline for a long time cannot be repaired by shipping the writes it missed if the log no longer contains them. The log is truncated, the segments are recycled, and the only remaining source of truth is the current state of a live replica. Copying that state is the bootstrap problem, and it is harder than it looks because a live replica’s files are not a state at any instant — they are a moving target that changes underneath the copy.

Why file-level copying produces a torn state

Consider a storage engine that keeps a data file and a write-ahead log. Copy the data file while writes are arriving and the copy contains pages from many different moments: page 40 as of the start of the copy, page 12 as of the end. The result is not a state the engine ever held, and replaying the log against it produces a database that is neither the old state nor the new one.

The failure is subtle because it is not a crash. Every byte in the copy is a valid byte that existed at some point. The invariant that is violated is the relationship between the pages and the log: the copy claims to be at log position p, but some pages have already absorbed changes from positions after p, and replaying from p applies them twice or applies them against pages that already reflect later changes.

Copying the log along with the data does not fix it either, because the two copies are taken at different times and the log may be truncated during the copy. The truncation is the more dangerous case: a log segment recycled while it is being read yields a file that is short or empty, and the copy silently contains a gap.

What a consistent cut actually requires

A consistent cut is a set of per-node states and in-flight messages such that every message received in the cut was also sent in the cut, as defined in the snapshot algorithm of Chandy and Lamport. For a single replica being copied, the requirement reduces to: the copy must correspond to one log position, and every change up to that position must be present in the data, with no change after it present.

The way to obtain that is to fix the position first, then take the data. The source replica records its current log position p, arranges for the data files to reflect exactly p, and copies them. Any writes that arrive after p are held in the log and shipped separately. The snapshot is then defined as “state at p”, and the log tail from p onward is the delta.

Two mechanisms are used to make the data reflect p. The first is a filesystem-level snapshot: the storage engine flushes and fsyncs, then the filesystem takes a copy-on-write snapshot of the data directory, and the copy is read from that snapshot while the live files continue to accept writes. The second is a block-level snapshot from the volume manager, which has the same effect one layer down. Both require the storage engine to have a notion of a flush point that corresponds to a log position, which is why the engine must cooperate even when the snapshot is taken below it.

Without such a mechanism the engine can still produce a consistent copy by holding a read lock for the duration, which blocks writes and is unacceptable for a large dataset. The snapshot approach exists to avoid that lock, and it is the reason copy-on-write filesystems and volume snapshots are commonly part of the backup path for replicated stores.

Stitching the snapshot to the stream

The snapshot must carry its position. The receiving replica starts from the snapshot’s state and then applies the log from position p + 1. If the sender’s log still contains p + 1, the receiver catches up incrementally and the transfer is cheap. If it does not, the receiver has a snapshot it cannot advance, and the bootstrap must restart from a newer snapshot.

This is why the log retention window is a bootstrap parameter and not just a recovery parameter. Raft makes the same requirement explicit: a leader that has discarded the entries a follower needs cannot bring it up to date incrementally, and must instead send a complete snapshot at a recorded index. A replica that takes longer to copy the snapshot than the sender retains log will finish the copy and then discover that the starting point is gone. The fix is either to retain more log than the maximum expected copy time, or to ship the log alongside the snapshot so the receiver has the segment it needs. The second is safer because it decouples the copy duration from the retention window, at the cost of transferring log that may be largely superseded.

A receiver that starts applying the log before the snapshot is complete is in the torn state described earlier. The order matters: snapshot first, fully, then log from the recorded position. Streaming the log into a separate directory and applying it only after the snapshot lands is the safe arrangement.

Incremental bootstrap and the difference problem

A full snapshot transfer costs the whole dataset. For a replica that is only slightly behind, that is wasteful, and the obvious improvement is to transfer only the difference.

The difficulty is finding the difference without comparing everything, which is the same problem anti-entropy solves with hash trees. A replica that has a stale copy can compare its per-range hashes against the source’s and request only the ranges that differ, which is precisely the Merkle tree descent applied to bootstrap. The limits are the same: the tree must correspond to a single position, and the transfer must be against that position rather than a moving target.

The incremental path is only valid if the receiver’s state is a consistent cut of the same history. A replica whose files were copied without a snapshot has an arbitrary mixture of positions and cannot be repaired by hashing ranges, because its notion of “what I have” does not correspond to any point in the log. Incremental repair assumes the starting state is meaningful, and the assumption is created by the original bootstrap, not by the repair.

Serving reads while catching up

A bootstrapping replica has a state at position p and is applying the tail. It is not up to date, but it holds a valid state at a known position. Whether it may serve reads depends on the consistency model, and the answer is usually yes with the caveat that the reads are stale by a known amount.

That known amount is the interesting part. The replica can report its applied position, and a client or coordinator can compare it to the leader’s current position to compute the lag in log bytes or entries. The conversion from log position to time is where honesty matters: bytes of log are not seconds, and the rate of log generation varies with the workload. A lag of a thousand entries is a fraction of a second under a heavy write load and minutes under a light one. Systems that expose lag in seconds are converting using a recent average, which is an estimate and should be labelled as one.

The alternative to reporting lag is to refuse reads until the replica is fully caught up, which is simpler and wastes the copy’s read capacity during the catch-up period. For a large dataset the catch-up can take long enough that the capacity matters, so the common choice is to serve reads with a documented staleness bound rather than to idle. The guarantee a client actually needs is usually per session, and the mechanism for enforcing it is described in session guarantees and read-your-writes.

The interaction with the durability contract

Bootstrapping does not change what the sender promises clients, but it does change what the receiver can promise. A replica created from a snapshot at p has no record of writes before p that it might need to serve, and no ability to participate in a decision about them. If the sender fails during the transfer and the receiver is the only surviving copy, the data after p is gone regardless of what was acknowledged to clients.

That is why bootstrap is unsafe on a cluster that is already at minimum redundancy. Starting a new replica while another is down means the snapshot source is the only complete copy, and its loss during the copy is unrecoverable. The operational rule is to bootstrap from a node that is not itself the last surviving copy, which is a statement about placement rather than about the copy mechanism — the redundancy arithmetic is in failure-domain placement.

There is also a durability question at the receiver. A snapshot transfer that is interrupted leaves partial files, and a receiver that resumes from them without a recorded position has a torn state. The safe pattern is to write into a temporary location, verify completion against the expected size and checksum, record the position, and only then move the files into place. Anything less turns an interrupted transfer into a replica that appears healthy and holds a state that never existed.

The commit point during a transfer

Bootstrap interacts with the commit point described in synchronous and asynchronous commit. A cluster configured to require acknowledgement from a specific replica cannot count the bootstrapping one until it is caught up, so adding a replica does not increase write availability until the copy completes. During the copy the cluster is running with one fewer acknowledgement target than the configuration suggests, which is exactly the window in which a second failure is most damaging.

The practical conclusion is that bootstrap is a redundancy-reducing operation for its duration, and that its duration is governed by dataset size, link bandwidth, and log generation rate. Scheduling it against a period of low write traffic reduces the log tail that must be shipped and shortens the window. What cannot be avoided is that the receiver is useless for durability until it holds a state at the current position, however fast the bytes move.