Producer Batching, Compression, and Acknowledgment Pipelining

A producer that sends one record per request pays for the network stack, the broker’s request parsing, a write, and at least one replication round trip, all to move a few hundred bytes. The overhead is not marginal: at small record sizes the per-request costs dominate the payload. Every producer design is therefore an exercise in amortizing fixed costs, and the three knobs that do it — a batch buffer, a linger deadline, and multiple requests in flight — each buy throughput with a different kind of latency.

Compression belongs in the same discussion because it only works at batch granularity. Compressing a single small record usually costs more than it saves; compressing a hundred similar records lets one dictionary serve all of them.

What a produce request costs before payload

Take a synchronous single-record send. The client serializes the record, builds a request, writes it to a socket, and waits. The broker reads the request, validates it, appends it to the partition log, and — if the topic requires acknowledgements from replicas — forwards it to followers and waits for their responses before replying. Only then does the client’s send call return.

The wall-clock cost is roughly one network round trip plus one disk write plus one replication round trip, and none of those three depend on whether the request carried one record or five hundred. A batch converts per-record costs into per-request costs. The larger the batch, the smaller the per-record share, up to the point where the broker rejects the request for size or the client’s memory for buffering becomes the limit.

Kafka exposes the two ends of this directly: batch.size caps the buffer for a single partition’s batch, sixteen kilobytes by default, and max.request.size caps a whole request at one megabyte. A record that exceeds the request limit fails locally, which is a deliberate constraint: a single oversized record would otherwise hold up the partition it belongs to.

Linger time is charged to every record

A batch fills only as fast as records arrive. Under light load, waiting for the batch to fill would mean waiting indefinitely, so the producer also has a time bound: linger.ms, five milliseconds by default in Kafka 4.0 and later and zero in earlier versions, after which a partially filled batch is sent anyway.

The interaction is a minimum, not an average. A record that arrives just after a batch was flushed waits for the next flush, which happens when the batch is full or the linger expires, whichever comes first. With a linger of five milliseconds and a full batch requiring a hundred milliseconds of arrivals, every record waits about five milliseconds and the batches are small. With a linger of fifty milliseconds, batches are larger and each record waits up to fifty.

The added latency is therefore bounded by the linger and paid on every record, not only on the ones that happen to be at the end. Under high load the batch fills before the linger expires and the setting stops mattering, which is the useful property: linger costs latency when traffic is light and nothing when traffic is heavy. Setting it to zero optimizes for latency and accepts whatever throughput the arrival rate allows.

A flush policy that respects both bounds looks like this; the interesting part is that the size check and the deadline check are independent.

import time

class BatchBuffer:
    def __init__(self, max_bytes=16_384, linger_s=0.005):
        self.max_bytes, self.linger_s = max_bytes, linger_s
        self.buf, self.nbytes, self.opened_at = [], 0, None

    def add(self, record: bytes):
        if self.opened_at is None:
            self.opened_at = time.monotonic()
        self.buf.append(record)
        self.nbytes += len(record)
        if self.nbytes >= self.max_bytes or self.expired():
            return self.flush()
        return None

    def expired(self) -> bool:
        # The sender's timer checks this for partitions with no arrivals.
        return (self.opened_at is not None
                and time.monotonic() - self.opened_at >= self.linger_s)

    def flush(self):
        batch, self.buf, self.nbytes, self.opened_at = self.buf, [], 0, None
        return b"".join(batch) if batch else None

The real client runs this per partition, not globally, because a batch is the unit that gets appended to one log at one offset.

Compression is a property of the batch

Kafka’s record batch format supports gzip, snappy, lz4, and zstd, and the codec applies to the batch payload rather than to individual records. The format’s header carries a CRC-32C over the batch, so a corrupted batch is rejected as a unit before any record in it is appended to the log.

Two consequences follow from batch-level compression. First, the ratio improves with batch size, because a larger input gives the compressor more context and a shared dictionary; a batch of nearly identical JSON documents compresses far better than the same documents sent separately. Second, decompression cost is paid once per batch on the consumer, which is why CPU cost per record falls as batches grow.

Codec choice is a trade between CPU and ratio. Gzip compresses well and costs the most CPU. LZ4 and snappy are much faster with modestly worse ratios. Zstd spans a wider range of settings and is usually the best default for high-volume streams where CPU is not the binding constraint. None of these statements is a benchmark; they describe the design position of each codec, and the right choice under a specific workload has to be measured on that workload.

Compression also changes what the broker stores and replicates. The compressed batch is what goes to disk and over the replication connection, so network and storage savings apply to the internal traffic as well as to the producer’s. The broker stores the compressed batch as received unless it has to recompress it, so the batch CRC is the integrity check that matters.

Acknowledgement modes and the replication bill

Kafka’s acks setting has three positions. acks=0 returns as soon as the bytes are written to the socket; the producer learns nothing, and a broker crash loses whatever was in flight. acks=1 waits for the leader to append, so a leader failure immediately after the reply can still lose the record, because followers had not yet copied it. acks=all waits for all in-sync replicas, which is the only setting that survives a leader failure without relying on timing.

The cost scales with the replication factor. With three replicas, acks=all means the leader waits for two follower acknowledgements, so the produce latency includes the slower of two network round trips plus the followers’ write time. That is the price of durability, and it is paid per request, which is another reason batching matters: the replication round trip is amortized across the whole batch.

acks=all alone does not guarantee durability. If the in-sync replica set shrinks to one because followers have fallen behind, acks=all is satisfied by the leader alone. The protection is min.insync.replicas, checked on the broker: a produce with acks=all fails if fewer than that many replicas are in sync. Setting it to two on a three-replica topic gives a majority, so a single failure cannot silently downgrade the guarantee.

Pipelining without reordering

A producer can have several requests in flight per connection, five by default in Kafka. This hides round-trip latency the way a sliding window does in TCP, and the throughput gain is substantial because the producer no longer waits for one acknowledgement before sending the next batch.

Retries break the simple version of this. If batch A fails and is retried while batch B succeeds, B is appended before A, and the log order differs from the send order. The idempotent producer fixes this by assigning sequence numbers per partition; the broker rejects an out-of-order sequence, and the client resends in order. The upshot is that in-flight pipelining and per-partition ordering coexist only with idempotence, or with the in-flight count reduced to one, which serializes the connection and costs exactly the round trips pipelining was meant to hide.

Ordering beyond a single partition is not preserved by any of this, a limitation examined in ordering guarantees and head-of-line blocking. The client-side buffer that makes batching possible is also a queue with a finite size, and its behaviour under a slow broker is the producer-side form of the window described in credit-based flow control and consumer prefetch limits.