Idempotency Keys and the Unknown-Outcome Problem

A client sends a request that creates a payment. The server commits the charge, then the connection drops before the response arrives. The client cannot distinguish this from a request that never reached the server, and its retry policy says to try again. Without some notion of operation identity, the second attempt creates a second charge. The window between “the effect happened” and “the client learned that it happened” is the unknown-outcome window, and no amount of transport-level reliability closes it. TCP retransmits, but a connection torn down by a load balancer after the backend committed is indistinguishable from a failure.

Idempotency keys are the standard remedy at the application layer. The client generates a unique token per logical operation and sends it with every attempt. The server records the token alongside the effect, so a repeat with the same token returns the original outcome instead of performing the work again. The result is at-most-once execution for a given key, which combined with at-least-once delivery from the client yields an effective once-only operation for the cases that matter.

What the key promises, precisely

The contract has three parts, and implementations differ mainly in which they honour.

First, the first request with a given key in a given scope executes the operation. Second, any subsequent request with the same key does not execute it again. It returns either the recorded response, or an error indicating that the first request is still in progress, or a conflict indicating the key was reused with a different payload. Third, the record survives long enough to cover the client’s retry horizon, and no longer.

The key is not a request identifier in the logging sense. It identifies an intended effect. Two requests with the same key and the same body are the same operation. Two requests with the same key and different bodies are a client bug, and the server should reject the second rather than silently apply the first client’s intent. Rejecting requires a fingerprint: a hash over the method, the target, and a canonical serialisation of the request body. Without one, a client that reuses a key after editing the payload receives a response describing an operation it no longer wants.

The header name is not standardised in an RFC. The IETF HTTPAPI working group has produced an Internet-Draft for an Idempotency-Key request header field, and several payment APIs use that spelling. Whatever the name, the semantics belong in the API documentation, because an intermediary cannot enforce them and a client that does not know the contract will not send the header.

The unknown-outcome window is wider than it looks

Client retries are only one source of duplicate requests. A load balancer with retry-on-502 configured will replay a request the backend already processed if the backend dies before responding. A service mesh with retry budgets can do the same one hop down. Long client timeouts chased by aggressive reconnects produce overlapping in-flight attempts. Each of these is a duplicate from the server’s perspective, and each is invisible to the code that wrote the original request.

That is why the key must be generated before the first attempt and reused across all attempts, including attempts made by infrastructure the client does not control. Generating a new key per HTTP attempt defeats the mechanism entirely: the retry looks like a new operation. The key belongs to the business action, so a “pay invoice” button generates one key when pressed and reuses it for every retry until the client receives a definitive answer.

For requests that are idempotent by method, such as PUT and DELETE, the key is redundant in principle, since repeating the request has the same effect. In practice a key is still useful for the response: a client that retries a PUT wants the final representation, not a 404 from a resource that was deleted in between. The mechanism generalises to PATCH and POST, where method semantics provide no protection at all.

Serialising duplicates in the database

The only reliable place to enforce the promise is a uniqueness constraint in the same transactional store as the effect. A cache or a Redis lock in front of the database reduces the number of duplicates that reach it, and cannot be the source of truth, because a lock can expire while the work is still running and a cache can lose the record after the effect committed.

The shape is a table keyed by the scope and the key, holding the fingerprint, a state, and the recorded response.

CREATE TABLE idempotency_records (
    account_id    bigint      NOT NULL,
    idem_key      text        NOT NULL,
    fingerprint   bytea       NOT NULL,
    state         text        NOT NULL,   -- 'in_progress' | 'completed'
    response_code integer,
    response_body jsonb,
    created_at    timestamptz NOT NULL DEFAULT now(),
    PRIMARY KEY (account_id, idem_key)
);

The first attempt claims the key with an insert. Under PostgreSQL, INSERT ... ON CONFLICT DO NOTHING RETURNING either returns the inserted row, meaning this attempt owns the operation, or returns nothing, meaning someone else claimed it.

INSERT INTO idempotency_records (account_id, idem_key, fingerprint, state)
VALUES ($1, $2, $3, 'in_progress')
ON CONFLICT (account_id, idem_key) DO NOTHING
RETURNING state;

When the insert returns nothing, the handler reads the existing row. If the fingerprint differs, it returns 422 or 409 with a body that names the mismatch, following the error model of the API. If the state is completed, it returns the recorded status and body. If the state is in_progress, the first attempt is still running, and the handler must wait, poll briefly, or return a conflict telling the client to retry later. Returning 409 for an in-flight duplicate is the simplest correct behaviour; a short bounded wait improves the client experience when the original is nearly done.

The critical requirement is that the claim and the business effect commit atomically, or that the claim is rolled back if the effect fails. If the claim commits and the effect does not, a retry finds in_progress and never executes. If the effect commits and the claim does not, a retry executes again. Wrapping both in one transaction works when the effect is a database write in the same database. When the effect is an external call, the two cannot share a transaction, and the design becomes a state machine with recovery: the record is written first as in_progress, the external call carries its own idempotency identifier that the provider honours, and a reconciliation process resolves records left in in_progress beyond a timeout.

Concurrent ownership needs a lease. A row stuck in in_progress because the process crashed must eventually be claimable, or the operation is permanently wedged. A started_at timestamp plus a takeover rule, where a second attempt may take over a record older than some multiple of the expected operation time, handles the crash case at the cost of allowing the original to complete concurrently if it was merely slow. That is a real double-execution window, and the mitigation is the downstream provider’s own idempotency, not a cleverer local lock.

Scoping, fingerprinting, and retention

The key must be scoped to a principal. A globally unique key namespace lets one tenant probe or replay another tenant’s response by guessing keys, and a 24-character token is not a secret. Scoping by account, and verifying that the authenticated caller owns the record, makes the replay safe. It also means two tenants can use the same key value without interfering, which matches how client libraries generate them.

Fingerprint canonicalisation needs a decision. A JSON body serialised with different key order, whitespace, or numeric formatting hashes differently even though it means the same thing. A canonical form, with sorted keys and a fixed number format, makes the fingerprint stable across client libraries. Hashes over the raw bytes are simpler and reject semantic duplicates, which is arguably the safer failure: a client that changes formatting gets a conflict and can generate a new key deliberately.

Retention is a trade between storage and safety. A window shorter than the client’s retry horizon lets a late retry execute a second time. A window measured in days keeps a growing table and a growing unique index, and every insert competes on the same index. The common shape is a time-based cleanup of completed records with a window aligned to the client SDK’s maximum retry duration, plus an index on the creation time to support the cleanup. Partitioning by day makes the deletion a partition drop rather than a scan.

The response itself must be stored, not just a success flag, or a replayed request returns a different body than the original. Storing a large response body for every mutation is a cost that catches people by surprise; storing a reference to a resource and reconstructing the response is cheaper when the response is derivable from current state, at the risk of returning a representation that has since changed.

Windows that stay open

The mechanism cannot make an operation exactly-once in the distributed-systems sense. It makes a specific client’s specific key at-most-once, and it depends on the client generating the key correctly and retrying with it. A client that crashes before persisting its key cannot retry safely, and a client that generates keys from a weak source can collide with another client’s key within the same scope, causing one operation to be silently suppressed. UUID version 4 gives 122 random bits, which makes accidental collision negligible within any realistic scope; a counter or a timestamp does not.

There is also the response-loss window on the client side. If the server records the operation and the client never receives the response, the client may give up permanently, leaving an effect the user believes did not happen. Reconciliation, a status endpoint keyed by the operation, or simply showing pending operations in the UI are the mitigations. Querying by key is a cheap addition to the contract: a GET on an operation endpoint returns the same recorded outcome, which lets a client that lost its connection check rather than guess.

Finally, the guarantee is per key, and a retry that omits the header falls back to POST semantics. That makes the header part of the API contract rather than an optional optimisation, and it justifies rejecting unsafe requests that lack one on endpoints where duplicates are expensive. Silently accepting them trades an immediate, fixable 400 for an occasional duplicate effect, which is the wrong direction: the duplicate stays invisible until someone reconciles a ledger, while the missing header is visible on the first request.