Acknowledgements, Offsets, and Visibility Timeouts

A consumer that has been handed a message is in an ambiguous state: it may be working, it may have finished, or it may be dead. The broker cannot distinguish these, so it relies on two mechanisms with very different failure behaviour. On a log, the consumer publishes a number describing how far it has read. On a queue, the broker grants a lease and takes the message back when the lease expires. The first is precise and late; the second is approximate and self-healing.

Getting either wrong produces the same symptom: work performed twice, or work never performed, with no error anywhere in the logs.

Position, committed offset, and the gap between them

A log consumer holds two numbers. The position is the offset it will fetch next, maintained locally and advanced as records arrive. The committed offset is the number it has published to the broker as durable progress. The gap between them is the amount of work that a restart will repeat.

Kafka’s consumer commits through commitSync or commitAsync, and can be configured to commit automatically at a fixed interval, five seconds by default under auto.commit.interval.ms. Automatic commits are convenient and slightly dangerous: the commit interval runs on the poll loop, so a commit can cover records that were returned by poll but not yet processed if the processing happens outside the loop. The offset that gets committed is the position after the last poll, not after the last completed unit of work.

The correct shape is to commit the offset after the effect is durable, and to commit exactly the offset that follows the last record whose effect is durable. Committing offset + 1 for a record at offset is a statement that everything up to and including that record is done. Committing the position returned by the most recent poll is a statement about what was delivered, which is a different claim.

There is a second timing constraint on the log side. A consumer that stops calling poll for longer than max.poll.interval.ms, five minutes by default in Kafka, is considered failed and its partitions are reassigned. Long-running work therefore has to be broken into records that fit inside that interval, or the group will rebalance mid-batch and hand the partition to someone else while the original consumer is still working.

Leases turn elapsed time into redelivery

A lease-based queue does not wait for a number. It hands out a message with a deadline attached. In SQS the deadline is the visibility timeout: thirty seconds by default, configurable per queue and per receive, up to twelve hours. While the timeout runs, the message is invisible to other consumers. If the message is deleted before it expires, it is gone. If not, it becomes visible again and another consumer may receive it.

The visibility timeout is a bet about processing duration. Set it shorter than the work and a second consumer starts on the same message while the first is still running, which multiplies load precisely when the system is slow. Set it longer and a crashed consumer’s message stays invisible for the whole timeout, so recovery takes that long. Neither value is right for a workload whose latency varies by orders of magnitude, which is why the lease is renewable: ChangeMessageVisibility resets the deadline for an in-flight message, letting a worker that knows it needs more time ask for it before expiry.

Renewal converts a fixed deadline into a heartbeat. The worker extends the lease at a fraction of the timeout, typically every third of it, and stops extending when it finishes or fails. The fraction matters less than the ordering: extend before expiry, and never extend after the work is done, or a message that was successfully processed will sit invisible until the extra time runs out.

import boto3

sqs = boto3.client("sqs")
queue = "https://sqs.example/queue"

resp = sqs.receive_message(
    QueueUrl=queue,
    MaxNumberOfMessages=10,
    WaitTimeSeconds=20,          # long poll; 20 s is the maximum
    VisibilityTimeout=60,
    AttributeNames=["ApproximateReceiveCount"],
)

for msg in resp.get("Messages", []):
    try:
        process(msg["Body"])
    except Exception:
        # Do not delete: let the lease expire so the message is retried.
        continue
    sqs.delete_message(QueueUrl=queue, ReceiptHandle=msg["ReceiptHandle"])

The ReceiptHandle is the lease token, not the message id. It changes on each receive, and a delete that uses a handle from an earlier receive succeeds without necessarily deleting the message, so a late delete is not a reliable way to remove the copy that another consumer now holds.

Acknowledgement batching widens the window

Acknowledging every record individually is the smallest window and the highest overhead: one round trip per record, and on a log one commit per record. Batching amortizes the round trips and widens the repeat window to the size of the batch. A batch of five hundred records acknowledged at the end means a crash at record four hundred and ninety-nine repeats all five hundred.

For a log consumer the granularity is the offset, so batching is inherent: a commit is a statement about a prefix of the log, and there is no way to say “all done except record 317”. That record blocks the commit of everything after it if strict prefix semantics are kept. Skipping it by committing past it silently loses the record; the alternatives are to retry it until it succeeds, to route it aside to a dead letter destination, or to accept that the partition stalls. The stall is often the correct choice for a short-lived error, and the wrong one for a poison message, a tension taken up in dead letter queues and poison message quarantine.

For a lease-based queue there is no prefix requirement, so a worker can acknowledge each message independently and let one bad message fail alone. That is a genuine advantage of the destructive-read model, and it is why work queues handle heterogeneous task durations more gracefully than a partitioned log.

The acknowledgement can be repeated, the effect cannot

Committing an offset twice is harmless. The broker treats the second attempt as a no-op: a commit that names an offset no higher than the stored one leaves the stored value unchanged. Deleting a message is tied to a receipt handle rather than to the message, so a repeat is only meaningful while that handle is the most recent one; the documentation states that a delete with an older handle may leave the message undeleted. The asymmetry that matters is between protocol state and business state: the acknowledgement is a statement about progress and can be repeated, while the business effect is a change and cannot.

A worker that is unsure whether its delete succeeded can call delete again with the same handle; the request is accepted, and the documented failure mode is a message that was not deleted, which redelivery will surface. A worker that is unsure whether its database write succeeded cannot repeat it without a guard. This is why the acknowledgement is cheap to make reliable and the effect is the part that needs design: the broker’s operations tolerate repetition, and the consumer’s do not.

What expiry does not fix

Leases handle crashed consumers but not slow ones. A worker that exceeds its lease without renewing loses the message while still holding it, so two workers run concurrently. If the work is not idempotent, that is a duplicate effect no amount of broker configuration prevents. The receive count attribute is the earliest warning: a message whose approximate receive count has risen is being retried, and a message whose count keeps rising is being retried for a reason.

The second failure mode is the opposite. A worker that crashes after deleting the message but before its effect is durable loses the work entirely, because deletion is irrevocable. The ordering rule is uniform across both models: make the effect durable first, acknowledge second. Reversing that order converts a duplicate into a loss, and losses are harder to detect because nothing is retried.

A third edge is the commit that succeeds while the process dies before it learns so. The retry commits the same offset again, which is harmless, and reprocesses the records after it, which is not. Offsets make the retry idempotent at the broker, never at the consumer, which is why the duplicate windows described in delivery semantics survive a perfect offset protocol.