Cutover Between Two Systems: Dual Writes, Shadow Reads, and the Switchover Point

Moving a workload from one datastore to another is not a copy operation. The copy is the easy part: read everything from the source, write it to the destination, and repeat until the difference is small. The hard part is that both stores are live while this happens, that clients are writing to one of them, and that at some instant the authority to answer reads must move from the source to the destination without any client observing an answer that the other store would contradict. The techniques that make this work (dual writes, shadow reads, and an atomic switchover) exist to close the intervals in which the two stores can disagree.

The interval problem

Suppose the source is authoritative. A client writes to it, the write succeeds, and a migration process copies the record to the destination some time later. During that delay the destination is missing the record, which is fine as long as nobody reads from the destination. The moment a reader is served from the destination, the delay becomes an inconsistency: the read returns an older state than the write acknowledged, violating the guarantee the client was given.

Three intervals produce this shape, and each needs its own mechanism.

The copy lag is the delay between a write landing in the source and appearing in the destination. It is bounded by the migration’s throughput and by how the copy is driven: a full scan that finishes and then stops leaves a growing tail, while a change stream that follows the scan keeps the lag bounded. Copying without a change feed produces a destination that is correct as of the scan’s start and progressively wrong thereafter.

The write skew window is the interval in which both stores accept writes. If a client writes to the source and another client writes to the destination, the two diverge with no ordering between them, and neither store can resolve the conflict because neither knows about the other’s write. Dual writes are the usual attempt to prevent this, and they do not, for the reason set out below.

The switchover instant is the moment reads begin to be served from the destination. If it is not a single instant from the perspective of every client, some clients read the new store while others read the old, and the two disagree for the duration of the overlap.

Why naive dual writes fail

The obvious design writes to both stores on every request, waits for both to acknowledge, and considers the write done. It fails in four ways, and the failures are not exotic.

The first is partial failure. The source write succeeds and the destination write times out. The client is told the write failed, but the source now contains it, and a retry may write it twice. Alternatively the client is told it succeeded and the destination is missing it. Either way the two stores disagree, and the disagreement is durable. No retry loop fixes it, because the retry is issued by a client that does not know which side succeeded.

The second is ordering. Two writes to the same record, issued concurrently through different application instances, can arrive at the two stores in different orders. Store A sees write 1 then write 2; store B sees write 2 then write 1. Both stores are internally consistent and permanently different, and no amount of comparing them identifies which order was correct.

The third is non-atomicity across stores. There is no two-phase commit between two independent systems unless both support it, and even when they do, the coordinator becomes a new failure domain and the commit blocks on the slower participant for every write. The latency cost of synchronous dual writes is the maximum of the two stores’ latencies, and the availability cost is the conjunction of their availabilities: if either is down, writes fail, even though one of them is fully capable of serving.

The fourth is that dual writes do not establish the destination as correct; they establish it as equally written. A bug in the translation layer that maps source records to destination records writes the wrong value to both, consistently, and the comparison that is supposed to validate the migration finds agreement.

// Dual write with a durable outbox instead of two synchronous calls.
func (s *Service) Write(ctx context.Context, rec Record) error {
    return s.source.Tx(ctx, func(tx Tx) error {
        if err := tx.Put(rec); err != nil {
            return err
        }
        // The intent to mirror is committed with the write, so a crash
        // cannot lose it. The relay retries until the mirror confirms.
        return tx.AppendOutbox(MirrorIntent{Key: rec.Key, Version: rec.Version})
    })
}

The outbox converts a cross-store atomicity problem into a local transaction plus an at-least-once relay, which is the only shape that survives crashes. What it does not solve is ordering: the relay must deliver intents in per-key order, and the destination must reject an intent whose version is older than the one it already holds. Without that rejection, a retried intent from a stalled relay overwrites a newer value, and the destination regresses.

Shadow reads as evidence, not as a serving path

Before switching reads, the destination can be exercised without being trusted: a fraction of production reads is sent to both stores, the source’s answer is returned to the client, and the destination’s answer is compared. This is the only mechanism that tests the destination against real traffic rather than against a synthetic corpus, and its value comes entirely from what it does with the comparison.

The comparison must be defined on a canonical form, not on the raw records. Two stores that represent a timestamp with different precision, a numeric with a different scale, or a null as absent versus present will report mismatches that are artefacts of encoding. The canonicalisation has to be written once and applied to both sides, and its rules are part of the migration’s specification rather than an implementation detail; it is the same problem that makes range digest comparison require an order-independent, encoding-stable digest.

The mismatch rate needs a threshold and a classification, and the comparison itself is the range digest problem applied to a live read path rather than to a completed backfill. A small rate of mismatches concentrated on records written in the last few seconds is copy lag and is expected. A small rate spread uniformly across old records is a translation bug. A rate that grows over time is a relay that has fallen behind. Distinguishing those three requires recording, for each mismatch, the age of the record and whether it was recently written, which is more useful than a single aggregate mismatch count.

Shadow reads also cost latency and load. Sending every read to both stores doubles the read capacity required and makes the source’s latency the maximum of the two, unless the shadow call is issued in parallel and its result is not awaited. Parallel-and-ignore is the right shape: the client’s answer comes from the source alone, and the shadow result is compared asynchronously. The cost is that a shadow call that hangs does not affect the client but does consume a connection, so the shadow path needs its own concurrency limit and timeout that are independent of the serving path.

The switchover must be one epoch

The switch is usually implemented as a configuration change: a flag, a routing rule, a DNS entry, or a value in a service registry. Every one of those propagates asynchronously, and during propagation some clients read the new store while others read the old. The divergence in that window is not hypothetical; it is exactly the divergence the whole migration was designed to avoid.

The fix is to make the switch a value that every participant reads at the same logical point, and to make the old store reject writes that belong to the new epoch. A monotonic epoch number, incremented once at cutover, serves both purposes. Clients read the epoch from a single authoritative location rather than from a local cache with a stale-while-revalidate policy, and route accordingly. Writes to the old store carry the epoch, and the old store refuses any write whose epoch is not the one it is configured for, which prevents a client that has not yet observed the switch from writing to the store that is no longer authoritative.

The epoch also resolves the ordering problem in the other direction. A write that arrives at the destination before the cutover is legitimate only if it came through the relay; a write that arrives after the cutover must come from a client. Tagging each with its origin lets the destination apply the correct conflict rule rather than treating them as equivalent.

-- Cutover in one statement, so no client sees a half-switched state.
BEGIN;
UPDATE routing SET epoch = epoch + 1, authority = 'dest'
WHERE name = 'orders';
-- The source stops accepting client writes at the new epoch.
INSERT INTO source_guard (epoch, accept_writes) VALUES (currval('epoch'), false);
COMMIT;

Whether that statement can be executed atomically depends on where the routing state lives. When it lives in the same transactional store as the data, the switch and the guard commit together and the cutover is genuinely atomic. When it lives in a separate configuration system, the best available approximation is a two-phase sequence: stop client writes to the source first, drain the relay, then flip reads. Stopping writes first is safe because the destination is already receiving every write through the relay; the reverse order is not, because a read from the destination during the drain can miss a write that is still in flight.

After the switch: the tail and the decommission

The cutover does not end the migration. Three tails remain.

The first is in-flight writes: requests that began before the switch and complete after it, against the source. They must either be rejected by the epoch guard or be forwarded to the destination by the source itself. The second is the relay’s backlog: intents committed to the outbox but not yet delivered. The relay must be drained to zero before the destination is declared authoritative for reads, and the drain must be verified by comparison rather than by the relay’s own counter.

The third is the decommission, and it is where the risk concentrates. The old store must remain readable and unmodified for long enough that a rollback is still possible: a period measured in days, not minutes, because the failures that justify going back are often discovered by users rather than by monitoring. During that period the old store is a cold copy: writable by nobody, read by nobody in the serving path, but present. Only after the rollback window closes is it deleted, and the deletion is the first genuinely irreversible step in the whole migration.

The decommission is also where the migration’s cost becomes visible in the wrong direction. Keeping a full copy of a large dataset for a rollback window doubles storage cost for that window, and the temptation to shorten the window is strong. The decision should be made on the basis of how long a silent data error could remain undetected, which for a system with aggregate reports and downstream consumers is longer than for one whose users see every record immediately.

The properties that make cutover tractable

Cutover is safe when four things are true: writes have a single authoritative path at any instant; the two stores are reconciled by a mechanism that survives crashes and respects per-key order; the switch is a value read at one logical point rather than a broadcast; and the previous store remains available for a bounded but non-trivial period afterwards. Each of those is a design decision made before the migration starts. None of them can be added during the cutover itself, because the state at that moment is already the product of whatever mechanism was built. The work of the migration is therefore mostly in establishing those properties, and the switch is a small operation performed once they hold.