What a Successful Write Guarantees: Durability and Read-After-Write Consistency

Two different promises hidden inside one status code

When a PUT returns 200, the service has made two claims that are often conflated. The first is durability: the bytes are now stored in a way that survives the loss of the hardware that accepted them. The second is visibility: a subsequent read from any client will return those bytes. The first is a statement about failure tolerance over years; the second is a statement about ordering over milliseconds. A store can offer the first without the second, and several did for a long time.

Durability is quoted as an annual probability of not losing an object, and the familiar figure for the large providers is eleven nines, or 99.999999999%, an expected loss of one object in a hundred billion per year. That number is not a measurement of hardware reliability. It is a model: given a per-device annual failure rate, a per-device bit error rate, and an assumed rate at which the operator replaces failed units, the probability that more copies fail within the repair window than the scheme can tolerate. Change the assumptions and the figure moves. The honest reading is that the scheme tolerates the concurrent loss of several devices, and that repair happens on a timescale of hours rather than weeks.

Visibility is a different construction entirely, and it is the part that changed most recently in the public services.

Replication, quorums, and where the write becomes visible

A store that keeps three full copies on three machines must decide how many acknowledgements constitute success. Requiring all three makes the write durable against any single loss and visible to any reader that consults a live replica, but a single slow or unreachable replica stalls every write. Requiring two of three keeps writing during one failure, at the cost of a window in which a replica still holds the old value.

The classic formulation is that with N replicas, a write acknowledged by W and a read served by R overlap when R + W > N. Three replicas with W = 2 and R = 2 intersect in at least one node, so a reader that takes the highest version among the responses sees the newest acknowledged write. This is the model Dynamo described, and it comes with the obligation to resolve conflicting versions, either through vector clocks or through last-writer-wins, because overlapping quorums do not prevent two concurrent writers from both succeeding on disjoint majorities.

Cloud object stores mostly did not expose this knob. They instead separated the write path from the read path. A PUT is acknowledged after the data reaches a durable set of storage nodes, and the location of those nodes is recorded in a separate metadata service that maps key to a list of replicas. Reads consult that metadata service first. Consistency then depends on whether the metadata update is linearizable: if a reader’s lookup sees the new replica list, it fetches new bytes; if it sees the old list, it fetches old bytes. The metadata service becomes the arbiter of visibility, and its own replication protocol determines what readers observe.

Azure’s original blob design made this split explicit: a stream layer stored the bytes and a partition layer held the key-to-extent mapping, with the partition layer’s log structured so that a reader following the tail of the log saw writes in order. S3’s older behaviour followed a similar shape, and the documented guarantees reflected it: a PUT of a new key was immediately visible, while an overwrite or a delete could return stale data to some readers for a short period, and listings were eventually consistent.

The overwrite case is the hard one

Creating a new key is easy because there is no prior value to distinguish from a new one. The store allocates a fresh location, writes the extents, and publishes a metadata record. Any replica that has no record for that key simply has no record, and a read either finds the published record or returns “not found”; if it returns not-found for a key that was just created, that is a consistency failure, and it is why read-after-write for new objects was the first guarantee offered.

Overwriting an existing key breaks the symmetry. Now two valid answers exist, and the metadata service must ensure that every read path sees the newer one. If reads are routed to replicas selected by a stale mapping, a reader gets the previous bytes with a 200 and no indication that it is stale. Caches in front of the store amplify the problem, since an object retrieved through a CDN or an in-process cache has its own expiry.

The fix, rolled out by S3 in December 2020, was to make every read and every list strongly consistent. The implementation is not documented in detail, but the observable contract is: after a successful write acknowledgement, a subsequent read returns that version, and a subsequent list reflects it. A failed write leaves no partial state visible. Deletes behave the same way, with the caveat that the delete of a key that never existed still returns success.

That change removed an entire category of defensive code: the sleep-and-retry loop after writing a manifest, the “list twice and compare” check. It did not remove the need to think about ordering, because strong consistency is a guarantee about a completed operation, not about concurrent ones. Two writers that both succeed produce a last-writer-wins outcome determined by the service’s internal ordering, and the loser is not told. Serialising writers requires a conditional write, which is a separate mechanism.

Why listings lag even when reads do not

A GET addresses one key and can be answered from a single authoritative lookup. A list walks a range of the ordered index, and that index is maintained separately from the per-key records because it must be sorted by key bytes and paginated. Updating it in the same operation as the object write couples the write latency to the index’s replication, so implementations historically decoupled them.

The consequence is that a bucket can return the new bytes for GET reports/2026-03.csv while a LIST with prefix reports/ still omits that key. Code that discovers work by listing, then fetches what it finds, is safe under that model: it may miss a key for a while but never acts on a key whose bytes are not yet readable. Code that lists to decide whether a write already happened is not safe, because absence in a listing does not mean absence in the store. The standard repair is to make the authoritative check a HEAD on the specific key, which addresses a single key and therefore inherits read-after-write.

Strongly consistent listing is now the documented behaviour on the major services, but the older asymmetry still shows up in systems that layer their own index over the store. A table format that commits by writing a metadata file and then relies on listing to discover it, as described in atomic commit over object stores, is reimplementing exactly this problem one level up, and the fix is the same: an explicit pointer rather than a scan.

Failure modes that survive the guarantee

Durability is a per-object probability, and it assumes the object is still being stored under a policy that protects it. Three failure modes bypass it.

The first is the client-side mistake: a delete issued by a lifecycle rule, a script with the wrong prefix, or a PUT of an empty body overwriting a good object. Versioning converts an overwrite into an additional version and a delete into a delete marker, so the previous bytes remain addressable; the mechanics are in versioning and delete markers. Without versioning, an overwrite is unrecoverable and the eleven-nines figure is irrelevant.

The second is the durability of the key rather than the bytes. If the only record of a key’s existence is a manifest object that is itself a single object, losing that object loses the dataset even though every data object survives. The exposure is concentrated in a small number of small objects, which is why manifests are often replicated across regions rather than merely across devices.

The third is a read that is consistent but semantically wrong because the caller assumed more. A 200 from a replica in a secondary region, under an asynchronous replication policy, reflects the state at the time the operation was replicated. Read-after-write holds within the region that accepted the write; across regions it holds only up to the replication lag, which is covered in cross-region replication and its consistency window. The guarantee is scoped, and the scope is the thing to establish before relying on it.