Cross-Region Replication and the Window Between Two Copies

Two durability claims that are not the same claim

A bucket’s default replication keeps several copies of each object within one region, usually across separate buildings within a metropolitan area. That protects against the failure of a device, a rack, or a building. It does not protect against the failure of the region, which is a failure of power, network, or control plane spanning every building in it, and it does not protect against an operator error that deletes a bucket, since the deletion propagates to every copy the bucket owns.

Cross-region replication is the second claim: copies exist in a second region, in a bucket the first region does not control, so a regional outage or an accidental deletion in one leaves the other intact. The two claims are independent, and a design that needs both has to enable both. The first is a property of the storage class and the second is a property of a replication configuration.

The cost is that the second copy is not written by the same transaction as the first. The PUT is acknowledged in the source region after the source’s own replication completes; the destination is updated afterwards, by a background process that reads the source’s change log and writes the destination. The gap between those two events is the replication lag, and it is a property of the workload’s write rate, the object sizes, and the service’s capacity, not a constant that can be quoted.

What the lag does to a failover

Any failover that promotes the destination region reads the destination as of the last replicated state. Objects written within the lag window are absent, and an object written within the window whose bytes had not finished transferring may be absent or, on some services, present in a partially replicated state that the destination treats as unavailable until complete. A failover therefore loses a tail of writes whose length is the lag at the moment of the failure, and the failure itself is the moment when the lag is hardest to estimate, since the source region’s control plane may be the thing that failed, so the source cannot report how far it got.

The mitigation is to make the lag observable and to bound it. The services expose a metric for replication latency, and some offer a guarantee on it: S3 Replication Time Control states that 99.99% of objects replicate within 15 minutes, and GCS’s turbo replication offers a recovery point objective of 15 minutes for dual-region buckets. Those are documented targets under stated conditions, not measured values, and they are stated as percentages because the tail is the point: one object in ten thousand may take longer, and a workload that cannot lose any write cannot rely on an asynchronous copy as its only protection.

The complementary technique is to write the same object to both regions from the application, which converts the asynchronous copy into two synchronous writes that the application orders itself. That doubles the write cost and doubles the exposure to the slower region’s latency, and it does not make the pair atomic: a crash between the two writes leaves one region with the object and one without, and reconciliation is the application’s problem. The advantage is that the application knows exactly which writes completed where, which is information the service’s replication metric does not provide per object.

Deletes, versioning, and what replicates

A replication configuration is a filter plus a set of behaviours, and the behaviours are where the surprises are.

Delete markers replicate only if the configuration says so. A versioned source bucket, described in versioning and delete markers, produces a delete marker for every plain delete, and whether that marker appears in the destination decides what a failover serves. If markers replicate, the destination agrees with the source that the object is gone. If they do not, the destination keeps serving the last replicated version, which may be exactly what a backup-style copy is for, and may be a correctness problem for a copy meant to be a failover target.

Deleting a specific version is not replicated. Version-level deletes are a source-region operation, and the destination retains the corresponding version. Over time the destination accumulates versions the source has discarded, which inflates storage and, more importantly, means the destination’s version history diverges from the source’s. A restore from the destination after a failover returns a history that includes data the source had deliberately removed.

Replication is not retroactive. Enabling a rule replicates objects written after the rule takes effect, plus, on some services, an optional backfill of existing objects that must be requested explicitly. A bucket that has existed for years and gains a replication rule has an empty destination until the backfill completes, and the backfill is a bulk copy whose duration scales with the bucket’s size. Any failover plan written on the day the rule was enabled is a plan with no data behind it.

Replication also does not replicate lifecycle outcomes. A transition to an archival class in the source is not automatically applied to the destination, and a lifecycle rule in the destination is evaluated against the destination’s own object ages. Two buckets with the same rule set can therefore hold the same object in different classes, and the retrieval latency a failover inherits depends on which class the destination chose. The pricing consequences of that divergence are in storage tiers and minimum durations.

Reads that cross the boundary

The default read path is regional. A client reading from the destination bucket gets the destination’s state, which lags; a client reading from the source gets the source’s state. Nothing in the service presents the pair as one logical bucket with a single consistency point, so an application that reads from whichever region is nearer is reading two different views of the same data.

That is tolerable when objects are immutable and versioned by name, because a reader that resolves a name in one region reads a consistent object even if it is an older one. It is not tolerable when the reader’s correctness depends on seeing the newest write, which is why the pointer-based commit designs described in atomic commit over object stores treat the pointer’s replication as a separate decision. A pointer that replicates with a lag means a reader in the destination region resolves an older snapshot; it will read a consistent snapshot, just not the newest one. Whether that is acceptable is a question about the workload’s read-your-writes requirement, not about the storage service.

The read-after-write guarantee described in durability and read-after-write consistency is scoped to the region that accepted the write. Stating it without that scope is the most common way a replication design goes wrong: the application is correct in the source region and quietly stale in the destination, and the discrepancy only appears when traffic moves.

Sizing the destination and the egress bill

Every replicated byte crosses a region boundary, and inter-region transfer is billed per byte at a rate well above storage. For a workload that writes W bytes per month, the replication charge is W times the inter-region rate, and it is incurred whether or not the destination is ever read. A destination that exists solely for disaster recovery therefore has a cost proportional to write volume, with no offsetting read savings.

The destination bucket’s request rate is a second constraint. Replication writes arrive as ordinary writes against the destination’s keyspace, so they are subject to the same prefix partitioning limits as any other writer, described in request rate limits and prefix partitioning. A source bucket that spreads writes across hashed prefixes replicates into a destination that receives the same spread, which is the good case; a source bucket that concentrates writes on a timestamp prefix replicates that concentration, and the destination’s per-prefix limit becomes the throughput ceiling for the whole pipeline. This is one of the few places where a naming scheme chosen for the source has a direct effect on a bucket in another region.

There is also a versioning interaction worth pricing. If both buckets are versioned and delete markers replicate, an overwrite-heavy workload replicates every superseded version to the destination, where it is billed until the destination’s own noncurrent-version rules expire it. The destination’s storage can exceed the source’s if its expiry rules are looser, and the divergence grows with write frequency rather than with dataset size.

What a failover drill has to check

The checks that matter are the ones about state, not about connectivity. Whether the destination’s newest object matches the source’s, for a sample of keys across prefixes. Whether delete markers reached the destination, and whether a key deleted an hour before the drill is served by the destination. Whether the pointer object of any committed dataset is the version the destination believes it is, and whether the data files that pointer names are present. Whether the destination’s lifecycle rules have moved objects into a class that requires a restore before a read.

None of these are answered by a metric that reports replication is healthy. They are answered by comparing specific keys across the two regions and recording the answer, which is why the drill’s output is a list of divergences rather than a pass or fail. The list is the input to the recovery point objective the workload can honestly claim, and the number on that objective is the lag the workload observed at the moment it looked, not the number in the service’s documentation.