Append-Only Logs and the Queue Abstraction

A queue and a log both accept records in an order and hand them to readers. Nearly every operational difference between a message broker and a log-based broker follows from one decision: whether reading a record removes it. A queue treats delivery as consumption and keeps a ledger of what is still owed to whom. A log treats delivery as a position in a file and keeps nothing at all.

That difference is invisible in a hello-world publish and subscribe. It becomes the dominant design constraint as soon as fan-out, replay, and consumer count grow, because the two models scale along different axes.

What a destructive-read broker has to remember

In the classic queue model, a message exists until a consumer acknowledges it. The broker therefore tracks, per message, at least four things: whether it has been delivered, to which consumer, when the delivery lease expires, and how many times it has been tried. Amazon SQS makes the lease explicit as a visibility timeout: a received message becomes invisible to other consumers for a fixed interval, and reappears if it is not deleted before the interval elapses. AMQP 0-9-1 brokers take a different route to the same obligation, holding each unacknowledged delivery in an “unacked” set associated with the channel that received it, released by basic.ack, basic.nack, or basic.reject.

The per-message ledger is what makes a queue good at work distribution. Two consumers pulling from the same queue cannot receive the same message at the same time, because the broker has a row for that message saying it is in flight. Retry with a delay is a natural extension of the same row. Priority and selective consumption are also natural: the broker is already choosing which message to hand out next, so it can choose by priority, by deadline, or by routing key.

The cost is proportional to backlog times consumers. A million queued messages with ten thousand in flight is a million records of bookkeeping, and every one of them has to be durable enough to survive a broker restart without losing an acknowledged message or resurrecting a completed one. Acknowledged messages are then deleted, which in a disk-backed store means either rewriting pages or leaving tombstones for a compaction pass.

Fan-out is the first place this model strains. If five independent services each need every event, the queue has to hold five copies, because each copy is deleted when its own consumer finishes. Broker-level exchanges make this convenient, not free: a fanout exchange in AMQP routes one published message into five queues, and the storage and the bookkeeping are multiplied by five. Wide fan-out with slow consumers is therefore the expensive case, and retaining a full history for a consumer that arrives later is not expressible at all.

The log removes the ledger

A log-based broker writes each record once, at the end of a segment file, and assigns it a monotonically increasing offset. Kafka’s offset is a 64-bit integer per partition; it is not a message identifier but a position, and the broker exposes it as such. A consumer’s state is one number: the offset it will read next. Reading does not mutate the log, so the broker needs no per-message record of who has seen what.

Segments are the unit of deletion. A partition is a sequence of files, typically sized in the hundreds of megabytes to a gigabyte, and retention is enforced by deleting whole segments that have aged past the configured window, or by deleting segments from the front once the partition’s total size exceeds a quota. Nothing about this depends on consumer progress, which is why a consumer that stops for a week can resume at its old offset as long as the segment containing that offset still exists.

The consequences cascade:

• Fan-out storage cost is independent of reader count. Ten consumer groups reading the same partition read the same bytes. The broker serves additional readers from the same files, so the marginal storage cost is nothing; what grows is network bandwidth and per-request work, not another copy of the data on disk.
• Replay needs no extra copy. A new consumer starts at the beginning of the retained window, or at any offset it likes, because the records are still there.
• Consumer position is external. Offsets are usually committed back to the broker or to a separate store so that a restarted process can resume, but that is a convenience, not the mechanism. A consumer that tracks positions itself is fully supported.

Offsets also give a cheap way to reason about lag: the difference between the log’s end offset and a group’s committed offset is the number of records that group has not yet processed. That single subtraction replaces the “approximate message count” that queue brokers must compute by scanning.

A minimal version of the storage model makes the asymmetry concrete. Offsets below are byte positions in a segment, which is how a reader can seek without an index of every record.

import os, struct

class Segment:
    """Append-only record file. The offset of a record is its byte position."""
    def __init__(self, path):
        self.f = open(path, 'ab+')
        self.f.seek(0, os.SEEK_END)
        self.end = self.f.tell()

    def append(self, payload: bytes) -> int:
        start = self.end
        self.f.write(struct.pack('>I', len(payload)) + payload)
        self.f.flush()
        self.end = self.f.tell()
        return start

    def read_at(self, offset: int) -> bytes:
        self.f.seek(offset)
        (length,) = struct.unpack('>I', self.f.read(4))
        return self.f.read(length)

There is no delete(offset) method, and its absence is the point. Deleting a record would invalidate every offset after it, and offsets are the interface.

Where the queue model still wins

A log gives up three things that queues do well.

Selective removal. A log cannot delete one record in the middle. If one tenant’s data must be erased on request while the rest is retained, a log cannot do it without rewriting the partition. Queues delete per message as a matter of course, so per-item lifecycle is trivial there and expensive here.

Per-message delay and priority. A log is read in order, front to back. A delay queue that holds a message for thirty seconds before making it visible, or a priority queue that serves urgent work first, is native to a destructive-read broker and requires a separate mechanism on a log. Delayed delivery services exist for log-based systems, but they are additional components, not properties of the log.

Competing consumers without a partition discipline. On a log, parallel consumption requires that the log be split into partitions and that each partition be owned by one consumer at a time. That is a real restriction: the number of consumers in a group is capped by the partition count, and a slow record blocks everything behind it in that partition. A queue can hand the next message to whoever is free, which gives finer-grained load balancing across heterogeneous workers.

The ordering guarantees differ as a result, and they are easy to overstate in both directions. A queue with several consumers has no useful global order, though FIFO variants exist that promise order within a message group. A log has a total order within a partition and no order across partitions. Neither offers an order across the whole topic, and neither can, without serializing everything through one place.

Sizing the choice against the workload

The decision usually reduces to two questions: how many independent readers need the same record, and does anyone need to read a record again after acknowledging it.

If readers are few and each message has exactly one owner, a destructive-read queue matches the semantics with less machinery. If readers are many, or a new reader appears later and needs history, the log’s write-once model is the only one that does not multiply storage by reader count. The replay requirement is decisive because it is hard to add later: a queue cannot reconstruct messages it has already deleted, so any design that might need to reprocess must either keep a copy elsewhere or start on a log.

The middle ground is common and worth naming: a log as the durable source of truth, with derived queues for work that needs per-item retry, delay, or priority. In that arrangement the log carries the history and the fan-out, while the queues carry the short-lived scheduling state that a log expresses poorly. The boundary between the two is exactly the ledger described above: anything that needs a per-message row belongs on the queue side, and anything that needs to be read twice belongs on the log.

Both models have to answer the same next question, which is what “delivered” means when a consumer crashes between receiving a record and finishing with it. That is the subject of delivery semantics and of the acknowledgement bookkeeping that each model implements differently.