Exactly-Once Delivery Across System Boundaries

A service that must both change its database and notify the rest of the system faces a problem no broker can solve. The database commit and the publish are two separate operations on two separate systems, and there is no ordering of them that is safe. Commit first and the process can die before publishing, so the change exists and nobody is told. Publish first and the process can die before committing, so consumers act on a change that never happened.

Every technique that claims exactly-once across a boundary is a way of collapsing those two operations into one, or of making the second one repeatable without effect.

Why the naive orderings fail

Consider a service that inserts an order and publishes OrderCreated. If the insert commits and the publish fails, the order exists in the database with no downstream record of it, and no retry will ever send the event because nothing records that it is owed. If the publish succeeds and the insert rolls back, downstream systems have created a record for an order that does not exist, and no compensating logic knows to undo it.

Retrying around the failure does not help. A retry loop that publishes until it succeeds and then commits converts the first failure into the second: the publish succeeded, the commit failed, and the retry publishes again. The best it can achieve is at-least-once on the publish and exactly-once nowhere.

The root cause is that the two systems have independent commit points. Distributed transaction protocols exist to merge them, and they are discussed below; the reason they are rarely used is not that they fail to provide atomicity but that the atomicity they provide is expensive to operate and does not compose.

The outbox makes one commit carry both facts

The transactional outbox pattern removes the second system from the critical path. Instead of publishing to the broker inside the request, the service writes the message into a table in its own database, in the same transaction as the state change. One local commit now covers both the business row and the intent to publish.

A relay then reads the outbox and publishes. Because the intent is durable, a crash between commit and publish loses nothing: the relay picks up the row on its next pass. Because the relay may publish a row and crash before marking it sent, the publish can happen twice, which is why the pattern produces at-least-once delivery and requires the consumer to be idempotent.

CREATE TABLE outbox (
  id           bigserial PRIMARY KEY,
  aggregate_id text        NOT NULL,
  event_type   text        NOT NULL,
  payload      jsonb       NOT NULL,
  created_at   timestamptz NOT NULL DEFAULT now()
);

BEGIN;

-- Relay: claim a batch; other relay instances skip these rows.
SELECT id, aggregate_id, event_type, payload
FROM outbox
ORDER BY id
LIMIT 100
FOR UPDATE SKIP LOCKED;

-- Publish the claimed rows, then delete them before committing.
DELETE FROM outbox WHERE id = ANY ($1);

COMMIT;

SKIP LOCKED lets several relay processes share the table without contending on the same rows, and ordering by the sequence keeps a single relay’s publish order consistent with the commit order. The sequence is also the natural ordering key for downstream deduplication.

The design constraint is that the relay must publish in order per aggregate if consumers depend on per-aggregate order. A single relay reading in id order achieves that trivially and caps throughput at one process. Multiple relays partitioning by aggregate_id preserve per-aggregate order and scale horizontally, at the cost of not preserving a global order — which is usually the same guarantee the broker offers anyway.

Deleting outbox rows immediately after publishing keeps the table small and removes the audit trail. Keeping them allows a replay from the database, which is useful for rebuilding a downstream system, and requires a retention policy so the table does not grow without bound. Both are defensible; the choice should be made deliberately rather than by leaving the rows forever.

Log tailing replaces polling

Polling the outbox costs a query per interval even when nothing changed, and it adds the polling delay to every event’s latency. Change data capture moves the relay into the database’s write-ahead log: a connector reads the WAL, turns row changes into events, and publishes them. Latency drops to the replication delay, and the polling load disappears.

The trade is that the publication order is now the WAL order, which includes every table in the database, not just the outbox. Filtering by table is straightforward; filtering by business transaction requires care, because a transaction’s rows appear in the WAL in write order and may not be adjacent after filtering. A connector that emits per-row events turns one transaction into several events with no explicit grouping, so consumers that need the transaction as a unit must reconstruct it from markers or from a transaction id if the connector exposes one.

Log tailing also depends on a replication slot. The slot keeps WAL available for the connector, so a connector that stops for long enough pins WAL and grows the database’s disk usage; if the slot is dropped or invalidated, the connector loses its position and requires a resnapshot, which is a different operational failure from a stuck poller and needs its own alarm.

Broker transactions cover the broker only

Kafka’s transactions bind a set of producer writes and a set of offset commits into one atomic unit, with a transaction id that survives producer restarts and a coordinator that resolves in-flight transactions after a crash. A consumer with isolation.level=read_committed sees only records from committed transactions, and aborted records are filtered out rather than deleted.

This solves the read-process-write loop when both the input and the output live in Kafka: the input offset and the output records commit together, so a restart reprocesses exactly the records whose outputs did not commit. It also solves the producer retry duplicate through the idempotent producer’s sequence numbers.

It does not solve the boundary problem. If the consumer writes to an external database and then commits the offset, the database write is outside the transaction. A crash after the write and before the commit replays the record, and the write happens again. If the offset commit and the external write are instead coordinated by a distributed transaction protocol, the problem becomes the two-phase commit problem described next, with an in-doubt window that a coordinator must resolve.

Two-phase commit moves the ambiguity

A distributed transaction protocol coordinates a prepare phase, in which every participant durably records that it can commit, and a commit phase, in which every participant is told to do so. If all participants prepare successfully, the outcome is commit and the coordinator’s decision is durable before any participant is told. If any participant fails to prepare, all abort.

The failure mode is the in-doubt window. A participant that has prepared and then loses contact with the coordinator holds locks and cannot decide unilaterally: it must wait for the coordinator’s log to be replayed. That wait can last as long as the coordinator is unavailable, and it blocks every other transaction touching the same rows. This is why operational guidance usually discourages prepared transactions for high-volume paths, and why PostgreSQL ships with max_prepared_transactions set to zero, disabling the feature until an administrator enables it.

Two-phase commit is not wrong. It provides atomicity across heterogeneous resources where that atomicity is genuinely required, such as moving funds between two databases. It is a poor fit for publication, because publication does not need the broker and the database to commit together — it needs the intent to publish to survive a crash, which the outbox provides with one local transaction and no coordinator.

What deduplication can promise

Every design above reduces to the same consumer-side obligation: applying the same message twice must have the same effect as applying it once. Three properties make that possible.

A stable identifier that does not change across retries. Producer-assigned ids satisfy this; broker-assigned offsets do not, because a republished message gets a new offset.

A deduplication record written atomically with the effect. A unique constraint on the message id in the same transaction as the business change, as described in delivery semantics and duplicate windows, closes the window. A separate cache of seen ids does not: a crash between the effect and the cache write reopens it, and the cache’s eviction policy becomes a correctness parameter.

A retention window longer than the maximum redelivery interval. Deduplication is bounded in time. A message redelivered after the record has been pruned is applied again. The window must therefore exceed the longest plausible delay between first delivery and final retry, including quarantine periods and manual re-drive.

There is a fourth, less obvious property for systems with concurrent writers: fencing. If two instances can both process the same partition, an idempotency key alone does not prevent the stale one from writing. A monotonic version carried in the message, checked on write, rejects the older value. This is the same mechanism that makes last-write-wins safe only when the clock or the version is trustworthy.

Within these limits, exactly-once is achievable in a precise and useful sense: each message has one effect, even though it may be delivered many times. Across arbitrary boundaries, without cooperation from the consumer, it is not achievable at all, and the honest design target is at-least-once delivery with idempotent effects. The mechanisms that support replay and rebuilt state, described in log compaction, retention, and replay, assume exactly that contract.