Batching and the Amortization of Per-Request Cost

Almost every expensive operation in a backend has a fixed component that does not grow with the size of the work and a variable component that does. A disk write pays a seek and a command before it pays for bytes. A network call pays a round trip before it pays for payload. A database transaction pays a commit record and a flush before it pays for rows. Batching is the technique of putting several units of work into one such operation so that the fixed component is paid once. Its currency is throughput; its price is the time spent waiting for the batch to fill.

The trade is not free and it is not always favourable. Whether batching helps depends on the ratio of fixed to variable cost, on the arrival rate, and on the latency budget. Whether it hurts depends on the same quantities, plus one more: the batch’s internal serialization, which grows faster than the fixed cost it saves when the batch is large.

The fixed-cost model

Let a single operation cost F fixed plus V per item, and let a batch of size b cost F plus bV, with the batching itself free. The per-item cost is then F/b + V, which falls toward V as b grows. A disk with a fixed cost of a few hundred microseconds and a per-byte cost in the nanoseconds has a ratio of F/V in the hundreds for small items; a network call with a one-millisecond round trip and a payload transmitted in tens of microseconds has a ratio in the tens. The higher the ratio, the more there is to amortize and the smaller the batch needs to be to capture most of the gain.

The gain is asymptotic rather than linear. Going from b = 1 to b = 10 removes ninety percent of the fixed cost; going from 10 to 100 removes another nine percent of the original. The second increment costs ten times the waiting time for a tenth of the remaining benefit. This is why batch sizes cluster at small numbers even when the fixed cost dominates: the marginal return collapses quickly, and the latency cost does not.

// Group commit: one fsync for N transactions, one wait shared by all of them.
// Cost model: one flush plus N writes (F + N*V), against N*(F + V) when each commits alone.
static void group_commit(struct commit_queue *q) {
    size_t n = q->len;
    if (n == 0) return;
    writev(q->fd, q->iov, (int)n);          // one syscall for the whole batch
    fsync(q->fd);                      // the fixed cost, paid once
    for (size_t i = 0; i < n; i++)
        complete(q->items[i]);         // every waiter released together
    q->len = 0;
}

The fixed cost in that sketch is the flush. Its price is not the syscall but the device’s guarantee that the data is durable, which involves a cache flush and, on spinning media, a rotational delay. Grouping many commits behind one flush divides that price by the group size, and it is the reason write-ahead-logged databases can commit thousands of small transactions per second on hardware whose flush latency is measured in milliseconds.

The waiting cost and where it is paid

The batch must fill before it is sent, and filling takes time proportional to the arrival rate. At λ arrivals per second and a batch of size b, the mean wait for the last item to arrive is roughly b/(2λ) if arrivals are uniform, and the batch’s total latency is that wait plus the batch’s own service time. For a batch of 100 at 1,000 arrivals per second, the wait is about 50 milliseconds, an order of magnitude more than the per-item service time the batching was meant to hide.

That wait is the entire cost of batching, and it is paid by every item in the batch, including the first one that arrived. A design that batches for throughput alone will choose a batch size that minimizes per-item cost and produces latency no interactive caller accepts. The correct formulation is a constrained optimization: choose the largest batch whose worst-case wait stays inside the latency budget, which usually means a time-based trigger as well as a size-based one. A batch that flushes when either b items have arrived or T milliseconds have elapsed bounds the wait at T, at the cost of sending partial batches when traffic is light.

The time trigger’s interaction with load is worth noting. At high load, the size trigger fires first and the wait is small; at low load, the time trigger fires and the batch is small, so the fixed cost is barely amortized. Batching therefore helps most when load is high, which is exactly when the system needs throughput, and helps least when load is low, when latency is cheap. The trade is self-correcting in that sense, but the low-load case still pays the time trigger’s latency for no throughput benefit, which is why some designs disable batching below a rate threshold.

Batching on the write path versus the read path

Write batching has a property that read batching lacks: the items are independent and their order within the batch does not matter for correctness, provided the batch is applied atomically or the individual items are individually acknowledged. That independence lets the batch be assembled by a separate stage, decoupled from the request that produced the item, which is how log-structured systems and message brokers achieve their throughput. The request is acknowledged when the item is queued, and the durability guarantee is provided by the batching stage’s flush.

Read batching cannot usually decouple the same way, because the caller is waiting for a specific result. Its form is instead the multi-get: one request that carries several keys and returns several values. The fixed cost saved is the round trip, and the variable cost is the per-key lookup, which does not shrink. Multi-get therefore helps when the round trip dominates the lookup, as when a cache client fetches twenty keys from a remote cache, and helps little when the lookup dominates, as with a local index probe.

The distinction matters for the failure mode. A write batch that fails must be retried as a whole or split, and a partial application leaves the batch in a state the caller did not request. A read batch that fails is retried as a whole with no such ambiguity. Write batching therefore needs the batch to be idempotent or the retry to be aware of partial application, which is the constraint that makes group commit harder to implement than it looks.

When the batch becomes slower than the sum

The model above assumes the batch’s internal cost is linear in its size. Real batches have at least three sources of superlinearity.

Memory traffic is the first. A batch of b items must be held somewhere until it is sent, and the buffer’s working set competes with the rest of the system’s cache. Past the point where the buffer exceeds the last-level cache, each item’s assembly costs a memory round trip rather than a cache hit, and the per-item cost stops falling. The batch size at which this happens is a property of the hardware, not of the application, which is why batch sizes tuned on one machine class do not transfer.

Serialization is the second. If the batch is processed by one thread, whether sorted, deduplicated, or validated, that processing is O(b log b) or O(b) with a large constant, and it runs while the items wait. The parallel portion of the batch’s processing is what batching was meant to exploit; the serial portion is a new bottleneck, described in contention ceilings and the limits of parallelism.

Tail amplification within the batch is the third, and it is the one most often missed. A batch completes when its slowest item completes, so the batch’s latency is the maximum of its items’ latencies. If one item in a thousand is slow, a batch of a thousand items contains a slow one with probability near one, and every item in the batch waits for it. Batching converts an occasional slow request into a routinely slow batch, which is the same maximum-of-many effect that fan-out produces, as described in fan-out and the amplification of tail latency.

Batch size, queue depth, and the feedback loop

A batching stage is a queue, and its batch size interacts with its queue depth. If the stage drains b items per operation and items arrive faster than it drains, the queue grows; if it drains faster than they arrive, the stage idles. The steady state has the stage’s throughput equal to the arrival rate, and the queue depth is whatever the difference integrates to during bursts.

The interaction with the caller’s timeouts is where batching becomes dangerous. A batch that takes longer than the caller’s timeout causes the caller to retry, and the retry adds another item to the same queue. Under overload, the queue grows, the batch takes longer, more callers time out, more retries arrive, and the stage’s throughput does not increase because it was already saturated. The batch size that was tuned for throughput becomes a liability, because it holds items longer precisely when the queue is longest.

The mitigation is to shrink the batch as the queue grows, the opposite of the intuition that a busy stage should batch more aggressively. A stage that adapts its batch size to keep its queue depth bounded is doing admission control in disguise, and it needs the queue depth to be measurable at the point where the decision is made. A fixed batch size with a fixed timeout is simpler and fails in the direction described above; an adaptive one fails less but is harder to reason about.

Where batching does not apply

Batching requires that the work be deferrable. A request whose response must be produced before the caller can proceed cannot be batched unless the caller is willing to wait, and interactive callers usually are not. The techniques that appear to batch interactive work, namely pipelining, multi-get, and coalescing identical requests, are all ways of making one round trip carry several items without deferring any of them past the point where the caller needs the answer.

Batching also requires that the items be independent enough to be grouped. A sequence of operations with data dependencies cannot be reordered into a batch without changing semantics, and a batch that respects the dependencies degenerates into a pipeline whose stages are the batches. That structure is still useful, since it is how a compiler’s passes or a query plan’s operators work, but its latency is the sum of the stages’ latencies rather than the amortized cost of a single stage, and the throughput gain comes from overlap rather than from amortization.

The honest summary is that batching is a throughput technique with a latency price that is paid by every item, and its benefit is bounded by the fixed-cost ratio and by the batch’s internal serialization. It belongs on paths where the fixed cost is large, the latency budget is generous, and the queue depth can be bounded; elsewhere it is a way to make a fast system slow in exchange for a number on a throughput chart.