Dual Writes, Replication Lag, and Read-Your-Writes Through a Cache

A write that touches a cache and a database is two writes to two systems that do not share a transaction. There is no commit protocol between them, no isolation level spanning both, and no rollback that undoes a cache mutation. Every guarantee about what a reader sees afterwards has to be reconstructed from ordering rules, version metadata, or session state. That reconstruction is the substance of cache consistency, and it is more productive to reason about it as a set of specific anomalies than as a single property called “consistency”.

Two writes and no shared commit

The database write and the cache mutation can be ordered in two ways, and each ordering has a failure mode that the other does not.

Write to the database, then invalidate the cache. If the invalidation is lost — the process dies between the two, the cache is unreachable, the message is dropped — the cache keeps serving the old value until its TTL expires. This is the safe failure: it delays convergence but never publishes a value that the database never held.

Invalidate the cache, then write to the database. If the database write fails, the cache has been emptied for no reason and the next read repopulates the old value from the database — a transient miss, not a corruption. But a concurrent reader can repopulate between the invalidation and the commit, and then the cache holds the pre-write value with a fresh TTL, which is worse than the first case because the entry looks new.

Neither ordering is safe against the interleaving where a read’s load straddles the commit. That window is opened by the populate step of cache-aside and read-through, and it is the race described under invalidation by deletion and versioned keys. Here the point is narrower: the ordering choice only determines which failure is likely, not which failures exist.

When the cache is populated on write rather than invalidated, a third ordering appears: database write, then cache set. Two writers can commit in one order and set in the other, leaving the cache holding the older value while the database holds the newer one. Unlike the delete case, this is not self-healing: both writes succeeded, so no later read corrects the entry. Update-on-write is only safe when the cache set is conditional on a version, or when writes to a key are serialised by something stronger than the cache’s own ordering.

Read-your-writes without a transaction

The guarantee most applications actually need is not global consistency but session consistency: after a client’s own write, subsequent reads by that client reflect it. Global consistency through a cache requires invalidating every reader’s copy; session consistency requires only that the writing client’s reads bypass or correct the stale copy.

Three mechanisms provide it, and they compose.

Route the writing client’s reads to the primary for a bounded period after the write. This is a routing rule, not a cache mechanism, and its cost is a temporary loss of read scaling for that client. The bound has to be derived from the maximum replication lag, which is the same quantity that makes lag-based routing fragile, since lag is not bounded under load.

Carry the version with the client. The write response includes the new version, the client sends it on subsequent reads, and the read path compares it with the version stamped in the cache entry. If the cached entry is older than the client’s version, it is treated as a miss and loaded from a source that can serve at least that version. The comparison is one integer compare per read; the hard part is the source-side guarantee that a replica is caught up to a given version, which requires the replica to expose its applied position — a log sequence number, a commit timestamp, or an equivalent watermark.

Write through the cache on the mutating path, and only for keys the writer owns. If the first read after a write is by the same client and the cache entry was updated synchronously with the database commit, the read hits the new value. A per-user cache can follow the same rule: the owner’s view is updated synchronously on write while other copies are left to invalidate. The constraint is that every writer of that key must follow the rule, and that a failed cache update must be repaired by a later read rather than assumed to have happened.

Replication lag turns a cache into a stale fast path

When the source is itself replicated, the cache sits in front of a replica pool and its misses land on whichever replica the load balancer chooses. Two replicas at different positions in the log can answer the same miss differently, so a key that is expired, re-read, and re-populated twice in quick succession can end up holding an older value than one that was evicted earlier. The cache does not know the position of the replica that served the load, so it cannot record it.

Recording it is the fix: the entry stores the source’s applied position alongside the value, and a reader that needs a newer position treats the entry as a miss against a replica that is at least that far along. This turns the cache from a source of unbounded staleness into a source of bounded, checkable staleness. It requires the source to expose a per-replica position that means the same thing across replicas, which is true for a shared log and false for independent per-shard logs.

Without such metadata, the only bound is the TTL, and the TTL must be set against the worst-case lag. That is a poor trade: the TTL must be long enough to cover a lag spike, which makes it long enough to serve genuinely stale data in normal operation.

Monotonic reads and the reordering of observations

A weaker but more insidious anomaly is a client observing a value, then observing an older one. With a cache in the path this happens without any replication involved: a read hits the cache and sees v5, the entry expires, the next read goes to a replica that is behind and repopulates v4, and the third read hits the cache and sees v4. The client has moved backwards in time.

Session guarantees address this at the client boundary. Read-your-writes and monotonic reads rule out two specific anomalies: a session that cannot see its own write, and a session that moves backwards in time. They do not constrain the order in which other sessions’ writes become visible, so the session can still skip states the source passed through. Implementing monotonic reads requires the client to remember the highest version it has observed and require that any subsequent read return at least that version — again a comparison, again dependent on the source exposing a version. Where the client cannot be changed, the same effect is approximated by pinning a session to a replica or by not caching per-session reads at all.

Detecting rather than preventing

Some systems choose to make the inconsistency visible instead of eliminating it. A response can carry the version it was derived from, and the client can compare versions across responses to detect a regression and retry against a fresher path. The cost is a retry, which is cheaper than a wrong decision but only if the client knows what to do with the signal.

The other detection mechanism is at the source: if writes are appended to a log with sequence numbers, a cache entry can record the sequence number of the read that populated it, and a background check can compare cached positions against the log’s current position to find entries that are suspiciously old. That finds entries whose invalidation was lost, which is otherwise invisible until the TTL expires.

What cannot be fixed at this layer

A cache cannot provide atomic visibility across multiple keys. If a transaction writes two entities and a reader observes both through the cache, the entries expire independently, and the reader can see one new and one old. Fixing that requires the read path to fetch both from one snapshot, which means either bypassing the cache for the pair or caching the pair under a single key that is written atomically — the standard remedy, and the reason composite cache keys exist. The same reasoning applies to a transaction that writes a row and a counter: the row is cacheable independently, the counter is not, and any derived total should be cached under a key that reflects the whole committed state.

The limits are those of the underlying source. A cache can preserve a source’s guarantee, and it can weaken one, but it cannot manufacture an ordering the source does not have.