What Exactly-Once Delivery Requires at Each Hop

A connector’s documentation says exactly-once, and a reader is entitled to ask exactly once with respect to what, measured where, and under which failures. The phrase is not meaningless, but it is a claim about a specific boundary, and most disputes about it come from two people describing different boundaries.

Unpacked, exactly-once delivery of an effect requires four conditions to hold simultaneously. Each is achievable in isolation. The conjunction is what makes the guarantee narrow.

The four conditions

A stable identity per record. Deduplication needs to know that two arrivals are the same record. A log offset is stable within a partition and changes if the record is re-partitioned or the topic is recreated. A primary key is stable only if the source never reuses it. A composite of key and version is stable if the version is monotonic. Without an identity that survives replay, there is nothing to deduplicate on, and the guarantee cannot start.

An idempotent write at the sink. Applying the same record twice must produce the same final state as applying it once. Insert-or-update keyed by identity is idempotent for the row; UPDATE t SET counter = counter + 1 is not, and no amount of position bookkeeping repairs it. A sink whose write is not idempotent requires the position and the write to be atomic, which is the next condition.

Atomicity between the position and the write. Either the position advances and the write lands, or neither does. This is the ordering problem in where the checkpoint goes, and the mechanisms that provide it are a shared transaction, a two-phase commit, or an idempotent write that makes the position redundant.

Cooperation from the source about what a record is. A source that delivers the same logical change under two different identities, whether because it was captured twice from two log positions or because an upstream retry generated a second row with a new key, defeats deduplication at the sink. This is where most real failures live, and it is the condition no sink can enforce.

Two-phase commit between processor and sink

When the sink is an external system with its own transactions and the processor has its own state, the atomic boundary is provided by a distributed transaction protocol. XA is the long-standing form: the processor prepares a transaction on the sink’s resource manager, records its own state, then commits both.

The protocol’s cost is well documented and structural. A prepared transaction holds locks on the sink for the duration of the window between prepare and commit, which is the time the coordinator needs to durably record its decision. If the coordinator dies in that window, the sink’s locks are held until the coordinator recovers, and if the coordinator never recovers, an operator must resolve the in-doubt transaction by hand. The window is normally milliseconds; the failure mode lasts as long as the outage.

Because of that, most streaming systems avoid XA and instead make the sink’s write carry enough information that replay is harmless. Kafka’s transactional producer is the notable alternative: the producer writes records and its consumed positions into the same log under one transaction, so a consumer reading that log sees either all of the outputs and the position advance or none of them. The atomicity comes from the log being the single durable medium for both, not from a distributed protocol between two systems.

The pattern generalises: exactly-once is cheap when the position and the data live in the same store, and expensive when they do not. A pipeline whose source and sink are the same database gets the guarantee from the database’s own transactions. A pipeline spanning a message broker and a relational database needs either XA or an idempotent sink.

Idempotence as a substitute for atomicity

If the sink write is idempotent, the position no longer has to be atomic with it. The consumer can write, then advance the position, and a crash in between simply replays the record into a sink that produces the same state. The position becomes an optimisation that bounds replay volume rather than a correctness requirement.

This is the cheapest form of exactly-once and the one most pipelines actually have, whether or not they say so. It requires:

• The sink enforces uniqueness on the identity. A database needs a primary key or unique index on the deduplication column, not merely an application-level check, because a check-then-insert races between concurrent writers.
• The write is an upsert or a conditional insert, not a blind insert. Blind inserts turn a duplicate into a constraint violation, which the consumer must then treat as success rather than as an error to retry, a distinction that is easy to get wrong and that turns a benign duplicate into a stalled pipeline.
• The identity is carried in the record. A record that lacks the source’s key cannot be deduplicated at the sink, and reconstructing it from the payload’s fields is only safe if those fields are themselves unique.

The upsert’s cost is write amplification: a duplicate that could have been rejected by a lookup instead rewrites a row, generating a new version, a new log record, and index maintenance. On a sink with heavy duplication, that cost is visible in write volume, and it is the price of not tracking the position atomically.

Where the guarantee terminates

Consider three hops: source log to staging table, staging table to transformed table, transformed table to an external API. Hop one can be exactly-once if the staging write is keyed and the position is in the same transaction. Hop two can be exactly-once under the same conditions. Hop three cannot, because the external API accepts a POST and has no idempotency key.

The pipeline is exactly-once through hop two and at-least-once at hop three. A caller that retries the API call after a timeout produces two effects, and no amount of upstream care prevents it. The only remedies are an idempotency key the API honours, or a reconciliation that detects and removes the duplicate afterwards.

This is why exactly-once is best described as a property of a boundary rather than of a pipeline. Each hop has its own, and the end-to-end property is the conjunction, which fails at the weakest hop. Documentation that states the guarantee for the whole pipeline is either describing a pipeline whose last hop is idempotent by construction, or is overstating.

The source-side condition is the one that breaks

Deduplication at the sink assumes one identity per logical change. Sources violate this in several ordinary ways.

A capture process that reads the same log twice, because it restarted from an earlier position after a crash, delivers the same change with the same identity, which deduplicates cleanly. That is the benign case, and it is the one idempotent sinks are designed for.

A source with two capture paths delivers the same change with different identities. A table captured both by a log reader and by a polling query produces the change twice under different keys if the polling path stamps rows with its own sequence. A connector that runs two instances without coordination, each reading a different partition, is fine; two instances reading the same partition produce two streams whose identities agree, so the duplicate is detectable.

A source that generates a new key on update defeats key-based deduplication entirely. A table whose primary key is regenerated by the application on each modification presents an update as a delete plus an insert with an unrelated key, and the sink sees two distinct records for one logical row. No sink-side mechanism can recover the relationship; the fix is upstream, in the source schema, which is why schema drift at the connector is not merely a type-mapping problem.

A source that reuses keys after deletion is worse. If row 4711 is deleted and a new logical entity is inserted with key 4711, a sink deduplicating on 4711 conflates two entities. The identity has to be (key, creation epoch) or a monotonically increasing surrogate, and the sink has to keep the tombstone long enough to distinguish them, which is what deletes and tombstones in append-only sinks has to arrange.

Testing the claim rather than reading it

A guarantee that cannot be falsified by an ordinary failure is not a useful statement. The way to establish what a pipeline actually provides is to enumerate the crash points and ask, for each, what the sink contains after restart.

Crash after read, before write: the record is re-read. Whether the sink sees it once or twice depends on whether the previous attempt’s write landed. With write-then-commit, it may land twice; with an idempotent sink, the second is absorbed.

Crash after write, before position advance: the record is re-read and re-applied. Idempotent sink absorbs it; non-idempotent sink double-applies.

Crash after position advance, before the next read: nothing is lost and nothing duplicated, provided the position advance was durable. If the position store buffered the advance and lost it, the record is re-read, which is the duplicate direction and therefore safe.

Crash during a batch: the batch is partially applied unless the sink’s transaction covers it. This is the case where a per-record commit and a per-batch commit differ, and where a partial batch with a single position advance can lose the unapplied records.

The enumeration produces a table of outcomes, and the pipeline’s real guarantee is the worst row of that table. A claim of exactly-once holds only if every row shows one effect per record, which requires the sink to be idempotent or the position to be atomic, and requires the batch’s transaction boundary to match the position’s granularity.

What to build first

Given the ordering of costs, the practical sequence is: make the sink keyed and idempotent first, because it makes every other failure benign; make the position advance follow the sink write, because it converts loss into duplication; then batch, because batching reduces position writes and increases duplicate volume in a bounded way. Distributed transactions come last, and are worth their cost only when the sink cannot be made idempotent and the duplicate volume is unacceptable.

The check that operates when the guarantee has silently failed at a hop nobody instrumented is reconciling a load for completeness.