Specifying Partial Failure, Atomicity, and Retry Semantics

A specification that describes only the successful path is a specification of a system that never fails. Real interfaces must state what a caller can conclude when a call returns an error, when it returns nothing at all, and when it is repeated. Three questions cover most of the ground: what state is the system in after a failure, is repeating the call safe, and can a caller tell whether a call that appeared to fail actually took effect.

The third question is the one that breaks naive designs. A caller whose request timed out cannot distinguish “the server never received it” from “the server applied it and the response was lost”. That distinction is not observable in general — it is the two-generals problem in its practical form — so any specification that requires the caller to distinguish it is unimplementable. What is implementable is making the ambiguity harmless, which requires the operation to be repeatable with the same effect.

The four outcomes a call can have

A careful specification enumerates outcomes rather than assuming two. For an operation that mutates state, the cases are: the operation did not take effect and no state changed; the operation took effect exactly once; the operation took effect and the caller does not know; and the operation partially took effect. The fourth is the one that atomicity is defined to exclude.

Atomicity is the claim that the fourth case cannot occur: from the perspective of any observer outside the operation, the state change is either entirely present or entirely absent. That claim is about observability, not about internal mechanics. A multi-step update with intermediate states visible to other transactions is not atomic even if it always finishes; a multi-step update whose intermediate states are invisible is atomic even if a crash forces recovery work at startup.

The distinction matters because atomicity is often specified as a property of the failure, when it is really a property of the concurrency. Two clients reading concurrently see either the before or the after state; that is the same guarantee that a crash-recovery protocol provides after restart, achieved by a different mechanism. A specification that says “atomic” without saying “with respect to which observers” leaves the isolation level unspecified, and isolation level is where most of the cost lives.

Idempotency requires a key the caller owns

An operation is idempotent when applying it twice has the same effect as applying it once. Setting a field to a value is idempotent; incrementing a counter is not; creating a resource with a server-assigned identifier is not, because the second call creates a second resource.

The standard repair is a caller-supplied key: the caller generates a value that identifies the logical operation, sends it with every attempt, and the server records which keys it has already applied. The key must be chosen by the caller rather than the server, because the server cannot tell a retry from a genuine second request — the payloads are identical by construction. Only the caller knows that these two transmissions are attempts at the same intent.

def apply_once(store, key, mutate):
    # store maps key -> recorded result. The key identifies the intent,
    # not the attempt, so a retry observes the first outcome.
    if key in store:
        return store[key]
    result = mutate()
    store[key] = result          # must commit atomically with mutate()
    return result

The comment on the assignment is the whole design. If the key is recorded in a separate transaction from the mutation, a crash between them leaves either an applied mutation with no record, which a retry duplicates, or a record with no mutation, which a retry silently skips. The record and the effect have to commit together, which means the key store is part of the same transactional domain as the data. That constraint is easy to miss when the key store is a cache or a separate service, and it is the reason idempotency implementations that use a cache in front of the database are only correct when the cache is authoritative for the duration of the window.

The window itself is a specification parameter. Keys cannot be retained forever, so the guarantee is “at most one effect per key within the retention period”. A retry that arrives after expiry is a new operation. Specifications that omit the retention period promise more than the implementation can deliver; specifications that state it convert an unbounded requirement into an operational parameter that can be sized against the retry policy.

The HTTP semantics specification handles the same problem at the protocol level by classifying methods as idempotent or not: RFC 9110 permits a client to retry automatically when the method is idempotent, and advises against automatic retry for a method that is not, unless the client knows the particular request is safe to repeat. The classification is a property of the method’s defined semantics, not of any particular handler, which is why PUT and DELETE are safe to repeat while POST is not — and why an application that wants retryable creation has to define its own key mechanism, since the protocol has none for that method.

At-least-once delivery and the deduplication obligation

Message transports generally offer one of three guarantees: at most once, at least once, or exactly once. The first loses messages on failure. The third cannot be provided end-to-end across a network without the receiver participating, because the sender cannot know whether the last acknowledgement was the one that got lost. What transports actually provide is at-least-once delivery plus enough metadata for the receiver to deduplicate, and the receiver’s deduplication is what makes the composite effect exactly-once.

This is a specification obligation, not an implementation detail. A consumer that is documented as receiving each message once, when the transport delivers duplicates, has a contract that cannot be honoured. The honest contract says: the consumer may receive the same message more than once, and must therefore be written so that reprocessing is harmless. That sentence changes the design of the consumer — it forces a processed-message table, or a natural key that makes the update idempotent, or an ordering discipline that makes the duplicate a no-op.

Ordering is a separate axis and is often conflated with delivery count. At-least-once delivery does not imply any ordering, and a transport that preserves per-partition order does not preserve global order. A specification that needs “the update is applied after the create” must therefore say which stream the ordering is relative to. When the create and the update can land in different partitions, the receiver has to handle the update arriving first, which means the message has to carry enough state to be applied out of order or be buffered until the prerequisite arrives.

Timeouts, and what a timeout actually asserts

A timeout is not a statement about the remote system. It is a statement about the caller’s willingness to wait. The remote operation may complete after the caller has given up, and it may complete before the caller’s request was even transmitted. A specification that treats a timeout as “the operation did not happen” is wrong in a way that produces duplicate side effects.

What a timeout does buy is a bound on resource consumption at the caller. The caller’s thread, connection, and buffer are released at the deadline, and the design question is whether the abandoned work at the remote end is cancelled or left to complete. Cancellation is not free: it requires the remote side to support an abort, and an abort that races with completion is itself ambiguous. Leaving the work to complete is the common choice, which makes the operation’s idempotency the only thing standing between a timeout and a duplicate.

The retry policy then has to be specified together with the idempotency guarantee, because the policy determines how many times the effect may be attempted and the guarantee determines how many times it may land. Retrying a non-idempotent operation with backoff is not a reliability measure; it is a duplication measure with a delay. The two belong in the same paragraph of the specification, and separating them is how systems acquire “we retry three times” as a documented behaviour alongside “duplicate orders occur occasionally” as an undocumented one.

Where the specification must stop

Some properties cannot be pushed onto the caller. If an operation has an external side effect that cannot be keyed — sending an email, charging a card through an interface that does not accept an idempotency key — then exactly-once is not achievable, and the specification should say so rather than implying it. The honest statement is a weaker one: the operation is attempted at most $n$ times, and the recipient may observe duplicates. Downstream consumers can then be designed to tolerate that, which is a requirement, not a defect.

The same reasoning applies to compensating actions. A saga that undoes a completed step is not the same as a transaction that never happened: the compensation is itself an operation with its own failure modes, and the intermediate state was visible. A specification that presents a saga as equivalent to an atomic transaction is claiming an isolation property the design does not have. Stating the visible intermediate states, and who may observe them, is the part that makes the design reviewable — and it connects directly to the frame conditions discussed in function contracts, which are the same claim at a smaller scale. Which failures are permitted at all is a refinement question: an implementation that invents an error the specification did not list has added traces the caller cannot handle.