Log Compaction, Retention, and Replay

A log that only ever grows is a disk filling up. Every log-based broker therefore has a deletion policy, and the choice of policy determines whether the log behaves like a transient event stream or like a durable table. Retention keeps a time window and discards the oldest segments. Compaction keeps the newest record for each key and discards everything superseded, indefinitely.

The two answer different questions. Retention asks “how far back can a consumer restart?” Compaction asks “what is the current value of each key, and can a new reader reconstruct it from the log alone?” A topic can use either, or both: Kafka’s cleanup.policy accepts compact and delete together. The difference is visible in the offsets a reader sees.

Retention removes prefixes in segment units

Kafka stores each partition as a sequence of segment files. Retention is evaluated per segment: a segment whose newest record is older than retention.ms, or whose partition’s total size exceeds retention.bytes, is deleted from the front of the log. Active segments are rolled at a size threshold so that deletion has granularity; a segment cannot be deleted in part without rewriting it.

The consequence is that retention is coarse. A one-hour retention with one-gigabyte segments deletes in one-gigabyte steps, so the effective retained window depends on write rate. A partition receiving little traffic may keep a segment far past its age limit because it is the active segment and still being appended to. Nothing is lost silently from a consumer’s point of view — the log start offset simply moves forward — but a consumer whose committed offset falls below the new start offset has no way back to the deleted records.

Retention therefore imposes a liveness requirement on consumers, and the requirement is easy to violate without noticing. A consumer group that is stopped for longer than the retention window wakes up with its committed offset pointing at a deleted segment. Kafka’s policy for this is configurable: reset to the earliest available offset, reset to the latest, or fail. Failing is the honest option when the consumer’s state depends on having seen every record; resetting to latest quietly skips an unknown gap.

Compaction rewrites the log into a key-value snapshot

Compaction applies a different rule: for each key, keep the record with the highest offset, drop the rest. The cleaner runs in the background. It builds an offset map by scanning the “dirty” portion of the log — the region that has accumulated enough new records to be worth cleaning — then copies live records into new segment files, skipping any record superseded by a later one for the same key.

Two properties of this process are subtle and both matter operationally.

First, offsets do not change and are not compacted away as numbers. A record that survives keeps its original offset, so after compaction the log contains a monotonically increasing but non-contiguous sequence. A reader may see offsets 4, 9, 12, 40 with nothing in between. Code that assumes contiguous offsets, or that treats offset + 1 as the next record, is wrong on a compacted topic; the next record is whatever the fetch returns.

Second, compaction never touches the active segment. Only closed segments are candidates, so the newest records for a key remain exactly where they were written. This bounds the cleaner’s work and guarantees that recent writes are never delayed by cleaning.

def compact(records: list[tuple[int, bytes, bytes]]) -> list[tuple[int, bytes, bytes]]:
    """Keep the highest-offset record per key, preserving surviving offsets."""
    latest: dict[bytes, int] = {}
    for offset, key, _ in records:
        latest[key] = offset          # later offset wins
    return [r for r in records if latest[r[1]] == r[0]]

The cleaner’s cost is proportional to the size of the dirty region, which is why it is throttled and why cleanup is not instantaneous. A topic that must expose current state within a bounded delay needs enough cleaning capacity for its write rate, or the dirty ratio grows and readers see multiple records per key for longer than expected.

Tombstones delete by writing

Compaction has no way to remove a key outright, because removing the last record for a key would be indistinguishable from never having written it, and a new reader would not know the key was deleted. The mechanism is a tombstone: a record with the key and a null value. The cleaner eventually discards all records for the key including the tombstone, but only after the tombstone has been retained long enough for consumers to see it. Kafka’s delete.retention.ms controls that grace period, with a default on the order of a day.

The grace period exists because a consumer that missed the tombstone would keep a deleted key in its local state forever. Any consumer rebuilding state from a compacted topic must therefore treat a null value as a delete, not as an empty value, and must be running often enough to observe tombstones before they disappear. The same retention-versus-liveness arithmetic applies to tombstones as to ordinary segments, with a shorter budget.

Replay is how state is rebuilt

The reason to keep history at all is that a consumer can reconstruct derived state by reading from an offset. A materialized view, a search index, or a cache can be built by replaying the topic from the beginning, and rebuilt from scratch after a bug corrupts it. This is the log’s principal advantage over a destructive-read queue, which cannot replay anything it has deleted.

Replay cost is the constraint. Rebuilding a view by reading a billion records is bounded by sequential read throughput and by the cost of applying each record, not by the size of the resulting state, which is why practical systems keep periodic snapshots of the derived state and replay only from the snapshot’s offset forward. The snapshot and the offset must be stored atomically, or a crash between them yields a state that is either ahead of or behind its recorded position.

Compaction changes the shape of replay in a useful way. Replaying a compacted topic yields at most one record per key in its cleaned segments, so the work is proportional to the number of distinct keys rather than to the number of events; the active segment can still hold several records for a key until it is rolled and cleaned. For a topic used as a changelog of a key-value store, that is the difference between replaying every update ever made and replaying the current contents. Retention, by contrast, replays a time window: enough to catch up a recent consumer, not enough to rebuild from zero once the window has passed.

Replay also re-executes side effects. A consumer that calls an external service during replay will call it again for every record in the window, which is the duplicate problem in its most concentrated form. Replay is only safe when the consumer’s effects are idempotent, keyed on the record’s identity, or suppressed entirely during rebuild. Rebuilding a cache is safe; rebuilding by re-sending emails is not.

Retention as a contract, not a setting

The operational reading of these policies is that they define what the log promises to its readers. A retention window of seven days promises that any consumer can be down for less than seven days and still resume without a gap. It promises nothing about a consumer down for eight. A compacted topic promises that the current value of every key is available, and says nothing about intermediate values, which may be cleaned at any time.

Those promises have to be matched by the consumers’ own guarantees. A group whose offset commits are stored outside the log must keep them consistent with the log’s retention; a group that resets on an expired offset has silently accepted data loss. The most common failure is not a broker bug but a mismatch: a retention window chosen for storage cost, and a consumer restart procedure that assumes unlimited history.

A second mismatch is between retention and the batch of work in flight. When a segment is deleted, a consumer reading it may receive an error and need to reposition; consumers that treat any fetch error as fatal will fail rather than resume. Handling that path is a small amount of code and a large amount of avoided incident, and it is closely related to the redelivery behaviour described in acknowledgements, offsets, and visibility timeouts.