Read-Your-Writes Across Replicated Session Stores

Replication is introduced for read capacity, for latency, or for surviving the loss of a region, and each of those reasons spreads session reads across machines that are not perfectly synchronized. Session state is unusually sensitive to that spread. A user who has just logged in, just changed a password, or just had a permission granted is the one client guaranteed to read the value that was most recently written, and is therefore the client most likely to observe a stale replica.

The guarantees that matter here have names from the replication literature. Monotonic reads promise that a client never observes older data after having observed newer data. Read-your-writes promises that a client’s own writes are visible to its subsequent reads. Writes-follow-reads promises that a write is ordered after the reads that causally preceded it. A session store needs the first two almost always, and the third whenever a read influences the write — loading a record, modifying one field, and storing it back.

The rotation that appears to undo itself

The canonical failure is small and reproducible. A login handler writes the new session to the primary and sends the new cookie. The next page load opens six connections; the load balancer sends four to a replica that has not yet applied the write. Those four requests find no record for the new key and treat the user as anonymous. The two that reached the primary see the session. The page renders as partially logged in, or redirects to the login page, and a retry a moment later works.

Nothing in this sequence is a bug in the application’s logic. It is a violation of read-your-writes caused by routing reads to replicas without a rule about which reads may be served there. The same shape appears whenever the write is a rotation, a revocation, a permission change, or a logout.

Time as a version, and why wall clocks fail

The simplest rule is temporal: after a client writes, route that client’s reads to the primary for a fixed interval — a second or two. To apply it, the read path needs to know that a write happened. Two carriers are practical. The cookie itself can hold a short-lived marker, set alongside the session cookie, indicating that reads must be primary-pinned until a timestamp. Or the session record can carry a monotonically increasing version, and the cookie can hold the version the client last saw.

Wall-clock comparison between the client’s marker and the server’s clock fails when clocks disagree, and they do: NTP holds hosts within a few milliseconds on a good network but can be off by much more across regions or in a virtualized environment where the host is paused. A timestamp comparison that is wrong in the permissive direction serves the stale read; wrong in the restrictive direction is merely slower.

Logical versions avoid the problem. The write returns an opaque token — a replication offset, a sequence number, or a term-and-index pair — and the read carries it as a precondition: serve only if this replica has applied up to that point, otherwise wait briefly or forward to the primary. Replication offsets are store-specific but cheap to plumb through the same cookie or an in-memory map keyed by session identifier. The cost is that the read path must be able to ask a replica where it is, which not every client library exposes.

// readSession returns the record only if the replica has caught up to need,
// otherwise it fails over to the primary instead of serving stale data.
func (s *Store) readSession(ctx context.Context, key string, need uint64) (Record, error) {
    for _, r := range s.replicas {
        if r.Applied() >= need {
            return r.Get(ctx, key)
        }
    }
    // No replica has caught up; a bounded wait here would be equivalent to
    // a slow request, so the primary is the cheaper answer.
    return s.primary.Get(ctx, key)
}

The need value is the offset returned by the write that created or modified this session, stored in the cookie alongside the identifier. It is not a timestamp, so clock skew does not enter the computation.

Owning a session instead of replicating it

An alternative removes the question by giving each session a single writer and a single reader. Sessions are sharded by identifier hash to one node, and that node is authoritative; other regions forward reads to it. Latency for a remote user becomes a cross-region round trip on every request that touches the record, which is on the order of tens of milliseconds within a continent and low hundreds between continents — enough to matter against a page with dozens of requests, which is why the usual refinement is a short-lived local cache with explicit invalidation.

Owning a session also gives conflict resolution for free. Two concurrent writes to the same session are serialized by the owner, and the last one wins in a well-defined order. Active-active replication does not give that: two regions can each accept a write to the same key, and merging them requires a rule. Last-writer-wins by timestamp silently discards one of them, and for a session record the discarded field may be the authentication timestamp or the permission set. Vector clocks, as used in Dynamo, detect the conflict but push the merge decision to the application, which for a session means deciding which of two login events is real.

Per-field merging is worse than it looks. Two fields updated independently merge cleanly only if the merge is commutative and the fields are disjoint; a version counter that both writers increment cannot be merged by taking either value. Counters need a CRDT-style structure — a grow-only counter per writer, summed on read — which is a large amount of machinery for a record that is usually written by one client at a time.

Failover and the writes that were in flight

Promoting a replica when the primary fails is where the guarantee becomes visible. If replication is asynchronous, the promoted node is missing some suffix of the primary’s writes, and every session created or rotated in that window disappears. Users who logged in seconds before the failure are logged out; users whose sessions were revoked in that window get their access back, which is the more serious direction.

Synchronous replication removes the loss at the cost of putting at least one network round trip on every session write. With a quorum — write to a majority of replicas before acknowledging — a promoted node that was part of the quorum has the write, and the system survives the loss of a minority. The latency cost is one round trip, issued to the quorum members in parallel and bounded by the slowest replica whose acknowledgement the write waits for, and session writes are infrequent enough (login, rotation, occasional attribute change) that the cost is usually acceptable where the correctness is not.

There is a subtlety in what “acknowledged” means for the client. If the login response is sent before the quorum write completes, the user holds a cookie for a record that may not survive. If it is sent after, the login takes as long as the slowest quorum member. Sending the response after the local write but before replication is the tempting middle, and it is exactly the configuration that loses sessions on failover. The trade is a latency budget for a durability guarantee, and the choice should be visible in the configuration rather than implied by a client library’s defaults.

Caching on top of replication

An application-level cache in front of the store reintroduces the same problem one layer up. A cache entry for a session key is valid until the TTL expires or an invalidation arrives, and an invalidation is a message that can be lost or delayed. Writing the new record and invalidating the cache are two operations that are not atomic together, so there is a window in which the cache holds the old value and the store holds the new one.

The ordering that narrows the window is delete-then-write for a cache-aside pattern, followed by a delayed second delete to catch a concurrent read that repopulated the old value. That second delete is a heuristic, and its correctness depends on the repopulation happening within the delay. For session data, a better option is to include the version in the cache key so a new version is a different key and the old entry dies by TTL. Key Design and Eviction in External Session Stores treats the store-side mechanics; the versioned-key trick applies identically to any cache in front of it.

What to route where

The routing rules that satisfy these guarantees are few and can be stated as a policy. Session writes go to the primary or to a quorum. Session reads immediately following a write for the same key go to the primary, or to a replica that has reached the write’s version. Reads for keys with no recent write may be served by any replica, which is the overwhelming majority of traffic. Revocation reads always go to the primary, because the cost of a stale answer is an unauthorized request rather than a repeated login.

Stated that way, the consistency requirement is narrow: it applies to a small set of keys for a short period after a write. The engineering work is in carrying the version from the write to the read, and in making the fallback to the primary a normal path rather than an error path. Systems that instead make every session read a primary read get the guarantee for free and pay for it in cross-region latency; systems that make every read a replica read get the latency and lose the guarantee. Load Balancer Session Affinity and Its Failure Modes covers the routing layer where that decision is often made implicitly.