Idempotency Keys and Duplicate Suppression at Service Boundaries

A client sends a request, the server commits the effect, and the response is lost to a timeout or a reset connection. The client cannot distinguish that outcome from a request that never arrived, so it retries. Without a mechanism at the boundary, the effect happens twice. The window is narrow, which makes the bug rare in normal operation and common during an incident, when timeouts fire in bulk and retries follow immediately.

Duplicate suppression is therefore a property of every write boundary, not an optional hardening step. It has two forms: operations defined so that repetition is naturally harmless, and operations that carry a key which lets the server recognise a repetition.

Why a retry cannot know what happened

The failure is a property of the network, not of either endpoint. A TCP connection that resets after the server wrote its commit tells the client nothing about whether the commit happened. Any client that retries after an ambiguous failure is choosing duplicates over losses, and losses are usually worse: a duplicated order can be cancelled, a lost order is a customer who believes they ordered.

RFC 9110 defines method idempotency as the property that multiple identical requests have the same effect as one. GET, HEAD, OPTIONS, TRACE, PUT, and DELETE are idempotent by definition. POST is not, and the specification is explicit that automatic retry of a non-idempotent method requires some out-of-band knowledge that the request is safe to repeat. Idempotency-key semantics are that out-of-band knowledge, expressed in the protocol rather than in a client’s assumptions.

Two designs satisfy the requirement. The first makes the operation intrinsically idempotent: PUT /orders/123/state with the full desired state, or a conditional write using If-Match with an entity tag so the server applies the change only if the client’s version is still current. The second keeps the operation a POST and attaches a client-generated key that identifies the intent, which the server records.

Intrinsic idempotency is cheaper and has fewer edge cases, but it does not fit operations whose meaning is “append this event”, “charge this amount”, or “send this message”, where the payload describes an increment rather than a state. Those are exactly the operations that matter most.

The claim record and its states

A key-based design stores a record per attempted operation. The record has four jobs: claim the key so that concurrent attempts cannot both proceed, remember the outcome so a retry can be answered identically, hold a fingerprint of the request so a key cannot be reused for different content, and expire so the table does not grow without bound.

The record moves through states. An attempt inserts a row in progress, performs the effect, and updates the row with the response status and body. A concurrent attempt that finds a row in progress must either wait for the first to finish or return a distinct “in progress” response. It must not proceed, and it must not report a 500, because both outcomes cause the client to retry into a request that is already running.

The unique constraint is the mechanism, not the check. Reading the table to see whether a key exists and then inserting is a check-then-act race with exactly the same shape as any other, and it fails under the concurrency that a retry storm produces.

CREATE TABLE idempotency_key (
    scope        text        NOT NULL,   -- tenant or client id
    key          text        NOT NULL,   -- client-generated, bounded length
    fingerprint  bytea       NOT NULL,   -- hash of method, path, canonical body
    state        text        NOT NULL,   -- 'in_progress' | 'done'
    status_code  integer,
    response     jsonb,
    created_at   timestamptz NOT NULL DEFAULT now(),
    PRIMARY KEY (scope, key)
);

-- Claim the key; zero rows returned means another attempt owns it.
INSERT INTO idempotency_key (scope, key, fingerprint, state)
VALUES ($1, $2, $3, 'in_progress')
ON CONFLICT (scope, key) DO NOTHING
RETURNING key;

The fingerprint comparison is what turns a collision into a rejection rather than a wrong answer. A client that reuses a key with a different body must receive a conflict, because applying the second body would violate the promise the key made. Stripe’s API documentation describes the same behaviour for its idempotency keys, including the requirement that a key be reused only with identical parameters.

Placing the claim in the same transaction as the effect

The difficult part of the design is not the table but the commit boundary. If the claim row is written in one transaction and the business effect in another, a crash between them produces either a completed claim with no effect, and the retry is answered with a success that never happened, or an effect with no claim, and the retry duplicates it. The pattern has been reduced to a dual write with extra steps.

The fix is to write both in one local transaction when the effect is a database change in the same store:

BEGIN;

-- The payment insert reads the claim CTE, so it runs only if the key
-- was free. Both statements commit together or neither does.
WITH claim AS (
    INSERT INTO idempotency_key (scope, key, fingerprint, state)
    VALUES ($1, $2, $3, 'done')
    ON CONFLICT (scope, key) DO NOTHING
    RETURNING key
)
INSERT INTO payment (id, order_id, amount_cents)
SELECT $4, $5, $6 FROM claim;

COMMIT;

The data-modifying CTE makes the dependency explicit: an empty claim produces an empty SELECT, so the payment row is never written. The response body for the replay still has to be stored separately, usually in the same transaction, because the amount is only known after the effect is computed.

When the effect is not a local database write, the transaction cannot cover it. Charging a card through a provider, sending an email, or calling a partner API all happen outside the local commit. In those cases the local transaction must record intent and the external call must be made by a relay that can retry, and the external provider must itself accept an idempotency key. The Dual-Write Problem and the Transactional Outbox describes the local half of that arrangement.

Retention, scope, and key shape

Three parameters decide whether the mechanism survives contact with client behaviour.

Retention must exceed the longest interval in which a duplicate can arrive. That interval is not the client’s retry timeout; it is the sum of the client’s retry window, any time it spends in a queue before retrying, and the delay of any store-and-forward layer between the client and the service. A supplier integration that batches failed requests and replays them overnight has a window measured in hours, and a deduplication table with a one-hour expiry will allow the replay through. A day is a common conservative floor, and multi-day retention is normal for financial operations.

Scope must include the client identity. Keys are client-generated strings and two tenants can pick the same one without coordination. A table keyed only on the client string turns tenant A’s key into tenant B’s duplicate, which is both a correctness bug and an information leak through the replay of a stored response.

Key length must be bounded. Clients send UUIDs, hashes, and occasionally a serialised request, and an unbounded text primary key on a large table has a real storage cost. Stripe documents a 255-character limit for its keys; whatever limit is chosen must be validated at the edge and rejected with a 400 rather than truncated, because truncation turns distinct keys into collisions.

The stored response is also part of the contract. A retry that receives the original status and body can proceed exactly as the first attempt would have. A retry that receives a bare 200 loses the resource identifier the client needed, and the client will then issue a lookup that has its own failure modes.

Where deduplication has to be business-aware

Key-based deduplication protects one hop. It does not protect a chain, and it does not know what a duplicate means.

HTTP deduplication at the edge does not cover messages that a consumer receives twice from a broker, because those messages carry no HTTP key. The consumer needs its own deduplication on a business identifier, usually the entity id plus a version or sequence number, and that check has to run inside the same transaction as the consumer’s write. A message offset is not a business identifier across topic rebuilds, and a message id assigned by the broker is often stable but not always preserved through replay tooling.

Duplicate detection on a natural key fails when the key is not stable across attempts. An order created with a generated id is safe; an order created with a timestamp and a random suffix in the id is not, because the retry generates a new one. This is why client-generated identifiers exist as a design choice rather than an implementation detail.

The final gap is the external effect with no key support. If a payment provider exposes no idempotency facility, no amount of local bookkeeping prevents a double charge after a timeout, because the local state cannot observe what the provider did. The available options are to reconcile against the provider’s records, to make the operation compensable with a refund, or to accept a small double-charge rate. The second option moves the problem into the workflow layer, where a failed step is undone by an explicit business action rather than by a deduplication key; Compensating Transactions in Multi-Service Workflows covers what that costs. Each of the three is a business decision, and none of them is an implementation detail that belongs to the boundary.