Replication Lag as the Real Recovery Point Bound

A recovery point objective is usually stated in minutes: “no more than five minutes of data may be lost.” Cross-region replication does not deliver that as a property. It delivers a stream, and the amount of data lost in a failover is whatever had not yet crossed the link when the primary stopped. That quantity is not a constant. It depends on write rate, on link capacity, on how the replication protocol batches, and on whether the lag is measured in bytes or in time — and the two measurements disagree exactly when it matters.

Two definitions of lag that diverge

The standard way to measure replication lag is time-based: the standby reports how many seconds behind the primary it is, computed by comparing the timestamp of the last applied record against the primary’s current clock. This number is easy to display and misleading under load.

Consider a primary producing 50 MB/s of write-ahead log and a cross-region link that sustains 40 MB/s. The standby falls behind at 10 MB/s and the time-based lag grows without bound, which is at least an honest signal. Now consider the same primary producing 5 MB/s against a link that sustains 40 MB/s. The link keeps up, the standby applies everything within a second, and the time-based lag reads near zero. But the bytes in flight at any moment are whatever the network buffered plus whatever the standby has queued, and at the instant the primary dies, those bytes are lost.

The second case is the dangerous one. A time-based lag of 0.2 seconds on a system writing 5 MB/s implies roughly 1 MB of unacknowledged data in the worst case, but the actual quantity depends on the protocol’s batching and acknowledgment granularity. If the primary acknowledges a client write as soon as it is in local durable storage and streams it asynchronously, then every write acknowledged in the last link-latency window is at risk. With a 40 ms one-way path, that window is 40 ms of writes plus the transmission time of whatever is queued. At 5 MB/s, 40 ms is 200 KB. At 50 MB/s, it is 2 MB.

So the honest statement of the RPO is: the amount of data acknowledged by the primary but not yet durably applied on the standby at the moment of failure. Time-based lag approximates that only when the write rate is stable and the link is not saturated.

Measuring the gap in the right units

Two measurements are worth keeping, and neither is the lag number on a dashboard.

Byte offset delta. Most log-shipping systems expose the primary’s current log position and the standby’s last applied position. The difference, in bytes, is the true volume at risk. It converts to an RPO time only by dividing by the write rate: 200 MB behind at 5 MB/s is 40 seconds of data; the same 200 MB behind at 50 MB/s is 4 seconds.

Last-acknowledged-transaction age. The most conservative measure is the age of the newest transaction the primary has acknowledged to a client and the standby has durably applied. That requires the protocol to carry a marker, but it is the quantity a recovery point objective actually describes. Without such a marker, the byte delta divided by recent average write rate is the best available estimate.

A dashboard that shows only seconds of lag will read healthy during a period when the write rate has just dropped and the backlog is large in bytes. The reverse also happens: a burst of small writes can make byte lag look trivial while the time lag is high because of per-record apply overhead on the standby.

Where the lag comes from

Four distinct sources contribute, and they respond to different fixes.

Propagation delay. The minimum is the one-way path latency, tens of milliseconds across continents. This is a floor, not a backlog: it does not grow, and it does not cause loss beyond the in-flight window.

Link bandwidth. If the log generation rate exceeds the link’s sustainable throughput, lag grows linearly and without bound. Compression helps when the log is compressible — row-oriented change streams often are, by a factor of two to four — and it costs CPU on both ends. This is the only source that produces unbounded growth, and it is the one to alarm on.

Apply throughput on the standby. A standby that must replay records through the same engine as the primary, with the same indexes and the same fsync cost per transaction, can fall behind even with an idle link. The classic case is a standby replaying a large batch of small transactions: the network delivers them faster than the standby can commit them individually. Batching the apply, or relaxing durability on the standby while it is not yet promoted, trades recovery point for replay speed.

Protocol acknowledgment granularity. Synchronous replication with synchronous_commit set to remote_write acknowledges after the standby has written the record but before it has flushed; on requires a flush; remote_apply requires the standby to have applied it so that queries see it. Each step costs a round trip and moves the RPO boundary. PostgreSQL documents these levels precisely, and the difference between them is exactly the difference between losing nothing and losing an unflushed buffer.

The fourth source is where the design choice lives. Everything else is capacity and tuning; acknowledgment level is a decision about what a commit means.

The synchronous middle ground and its cost

Synchronous cross-region replication removes the loss window at the price of putting the inter-region propagation floor into every commit. A commit that must wait for a standby 80 ms away costs at least 80 ms, and if the standby is down or partitioned, the primary must either block or degrade to asynchronous.

The standard resolution is quorum acknowledgment: commit is acknowledged when a majority of replicas have it, which allows one replica to be unavailable without blocking. With three replicas spread across regions, the majority is two, and the latency is that of the second-fastest replica, not the average. Placement therefore matters more than count: three replicas in three regions with one far away means the far one is never in the critical pair, and the commit latency is set by the two near ones. Quorum placement and its latency cost works through that geometry.

There is a subtlety about what a quorum commit guarantees for the RPO. If a write is acknowledged by a majority, then any future majority must intersect the set that acknowledged it, so the write survives the loss of any minority. That is the guarantee. It does not mean every replica has the data, so a failover to a lagging replica that then forms a new quorum can still lose recent writes until it catches up or is excluded. Promotion rules must therefore exclude replicas that are behind the acknowledged position, which is a constraint on which node may be promoted, not just on how many.

Degradation is a policy, not an accident

When the synchronous standby becomes unreachable, the primary has three options, and each maps to a different RPO.

# PostgreSQL: what a commit waits for
synchronous_commit = off          # local WAL write only; loss window = one WAL flush interval
synchronous_commit = local        # local flush; standby may be arbitrarily behind
synchronous_commit = remote_write # standby received, not flushed
synchronous_commit = on           # standby flushed to durable storage
synchronous_commit = remote_apply # standby applied; readers see it

The setting off is worth calling out because it is often enabled during incidents to restore throughput. It converts the durability guarantee from “committed means on disk” to “committed means in the operating system’s page cache”, and the loss window becomes whatever the kernel’s writeback interval is. That is a legitimate trade during a link failure, but it must be an explicit, logged decision with a defined end, not a setting that stays changed.

A timeout-based fallback is the common middle path: wait for the standby up to a bound, then proceed locally and record that the commit was not synchronously replicated. The bound must be shorter than the client’s own timeout, or the client gives up before the primary decides, and the write may commit after the client has already retried elsewhere. That double-write is a correctness problem, not just a latency one, and it is the subject of conflict resolution when two regions accept writes.

What the RPO statement should say

A defensible recovery point statement names four things: the replication mode in force, the acknowledgment level, the maximum tolerated byte lag before writes are refused or degraded, and the promotion rule that excludes lagging replicas. A statement like “RPO 5 minutes” without those is unfalsifiable, because any observed loss can be attributed to an unusual event.

The operational form is a threshold on the byte delta. If the delta exceeds a bound that corresponds to the tolerable loss at the current write rate, the primary stops accepting writes or switches to a mode with a stronger guarantee, and the incident is visible rather than silent. The bound has to be expressed in bytes because the write rate is what makes the time-based number move, and the bytes are what the link and the standby actually have to move.

The failure mode this prevents is the one where lag is monitored in seconds, the write rate falls during an incident, the seconds number looks fine, and the eventual failover loses far more data than the objective allowed. The seconds number was never the RPO. It was a proxy that happened to be accurate at the write rate the system was designed for.