Delivery Semantics and Where Duplicates Come From

The phrase “exactly-once delivery” is usually offered as a property of a broker, and it is almost never one. What brokers sell under that name is a combination of an idempotent producer, a transaction that ties an offset commit to an output write, and a consumer that only sees committed records. None of that prevents a message from arriving twice at the edge of the system. It prevents a duplicate from having a second observable effect inside a defined scope.

The way to make sense of the three standard labels is to place them at the point where a consumer has done its work and the broker has not yet learned about it. That gap is where every ambiguity lives.

The acknowledgement is a second message

Delivery is not one event. It is at least four: the broker writes the record; the record travels to the consumer; the consumer finishes its work; an acknowledgement travels back. The last leg is a normal network operation, and it can fail in two indistinguishable ways. Either the acknowledgement was lost, or the consumer died before sending it. From the broker’s side, both look like silence.

Silence forces a choice. The broker can assume the work was done and forget the message, or assume it was not and redeliver. There is no third option, because the broker has no way to inspect the consumer’s effects, and any attempt to ask (“did you process message 42?”) is itself a message that can be lost.

Assume success and the system is at-most-once: a message lost in the window between work and acknowledgement never comes back. That is a defensible choice for telemetry where a missing sample is harmless. Assume failure and the system is at-least-once: the message is redelivered, and a consumer that had in fact finished its work now does it twice. That is the default for work queues, and it is the reason duplicates are the normal case rather than an error.

MQTT makes the trade explicit in its QoS levels. QoS 0 sends and forgets, with no acknowledgement at all. QoS 1 is at-least-once: the receiver answers with PUBACK, and the sender retransmits until it arrives. QoS 2 is a four-step handshake — PUBLISH, PUBREC, PUBREL, PUBCOMP — which lets both sides agree that the message was received exactly once, at the cost of two round trips per message and state held on both ends until the exchange completes.

The MQTT case also shows what the guarantee covers. QoS 2 guarantees that the receiving client’s protocol stack hands the application the message once. It says nothing about the application’s side effect: if the application writes to a database and crashes before recording that it did, the stack has no opinion. Broker-level exactly-once protects the transfer, not the transaction.

Three windows where a duplicate is created

Duplicates have distinct sources, and it helps to separate them because each has a different fix.

Producer retransmission. A producer sends a record, the broker persists it, and the acknowledgement is lost. The producer retries and the broker appends the record again. If the producer has no identity for its records, the broker cannot tell a retry from a genuinely repeated value. Kafka’s idempotent producer addresses exactly this by assigning each producer a producer id and a per-partition sequence number, so the broker can discard a record whose sequence it has already stored. Without such a mechanism, the duplicate is created on the broker side and every downstream consumer sees it.

Redelivery after a consumer failure. The consumer receives, begins work, and dies. The broker’s lease expires, or the session closes, and the record is handed to another consumer. Whatever the first consumer had already committed to a database remains. This is the window that at-least-once semantics accept by design, and it cannot be closed by the broker at all, because the broker cannot roll back a third-party write.

Rebalance and offset-commit lag. A consumer group reassigns a partition while a consumer is mid-batch. Records processed but not yet reflected in a committed offset are processed again by the new owner. This one is interesting because it is not a failure of any single component: the system is working as specified, and the duplicate is the price of not holding a distributed lock across the batch.

The frequency matters for sizing. If acknowledgements are lost at rate p, and each loss produces exactly one duplicate, the duplicate rate is about p per record on the affected path. That is a rough model, not a measurement: real rates depend on how long leases are, how often processes restart, and how aggressive retry policies are. A system that tolerates a duplicate rate on the order of one in a million needs the same code as one that expects one in a thousand; only the cost of the deduplication differs.

What exactly-once can honestly mean

Three different things are sold under the name, and mixing them up produces designs that look correct and are not.

The first is the idempotent producer: the broker suppresses a retry that duplicates a record it already has. This eliminates the producer-side window and nothing else.

The second is transactional delivery: the broker ties a set of output records and a set of offset commits into one atomic unit. Kafka’s transactions do this: a consumer configured to read only committed records does not see the outputs of a transaction that failed, and the offsets it committed as part of that transaction advance only if the transaction committed. Read-process-write loops inside Kafka therefore get exactly-once semantics for their internal state.

The third is effectively-once processing: the consumer’s side effect is made idempotent or deduplicated by the consumer, so a second application of the same message changes nothing. This is the only form that extends past the broker boundary, and it is the only form available when the effect lands in a database the broker does not control.

The second does not imply the third. A transactional Kafka read-process-write loop that then calls an external payment API has an uncommitted side effect the moment the HTTP request succeeds and the transaction later aborts. The window has moved, not closed.

An idempotency check is usually a unique constraint on a message identifier, written in the same transaction as the effect:

BEGIN;

-- Claiming the id and applying the effect commit or roll back together.
WITH claim AS (
  INSERT INTO processed_messages (message_id, processed_at)
  VALUES ($1, now())
  ON CONFLICT (message_id) DO NOTHING
  RETURNING message_id
)
UPDATE accounts SET balance = balance - $2
WHERE id = $3
  AND EXISTS (SELECT 1 FROM claim);

COMMIT;

The constraint is the deduplication, and the shared transaction is what makes the check and the effect atomic. Splitting them into two transactions reintroduces the window: a crash between the two leaves a message marked as processed with no effect, which is worse than a duplicate.

Matching the guarantee to the effect

The choice is per effect, not per system. A single service often handles three kinds of message with three different answers.

Non-idempotent financial effects need deduplication keyed on a stable identifier that survives retries, plus a retention window at least as long as the maximum redelivery interval. Operations that are naturally idempotent — setting a field to an absolute value, deleting a row by key, publishing a state snapshot — need nothing. An increment by a message-supplied delta is commutative but not idempotent: applying the same delta twice changes the result, so it needs the same deduplication as a payment. Replacing += 1 with = value, where the value is the new total, removes the problem.

Two properties are worth stating as invariants. A message identifier must be assigned by the producer, not the broker, or a producer retry arrives with a new identifier and defeats deduplication. And the deduplication record must be written in the same durable transaction as the effect, or a crash replays the window the record was meant to close.

The mechanics that produce these windows are broker-specific: how long an unacknowledged message is held, when it becomes visible again, and what the consumer can do to control either. Those are covered in acknowledgements and visibility timeouts, and the partition-level coordination that produces the third window is covered in consumer groups and partition assignment.