Batching and the Amortization of Per-Item Cost

Writing ten thousand records one at a time to a file descriptor is the canonical slow operation, and its slowness has almost nothing to do with moving bytes. A syscall on a modern x86-64 machine costs on the order of a few hundred nanoseconds when nothing else intervenes: the transition into kernel mode, the argument validation, the context save, and the return. Ten thousand of them cost several milliseconds of pure overhead to write a total of a few tens of kilobytes. Writing the same data as one 40-kilobyte write costs one syscall plus whatever the page cache does with the buffer.

That ratio is the reason batching exists. Almost every system with a throughput-sensitive interface has a batch form and a single-item form, and the batch form is not a convenience. It is the only way to reach the throughput the underlying mechanism is capable of, because the fixed cost is per call and the useful work is per item.

Fixed cost, variable cost, and the ratio

The model is simple enough to write down exactly. An operation that handles $b$ items costs $c_{\text{fixed}} + b \cdot c_{\text{item}}$, so the per-item cost is $c_{\text{fixed}}/b + c_{\text{item}}$. At $b = 1$ the per-item cost is $c_{\text{fixed}} + c_{\text{item}}$. At $b = 100$ the fixed term has been divided by a hundred.

When $c_{\text{fixed}}$ is much larger than $c_{\text{item}}$, as with syscalls and network round trips, the gain is nearly linear in $b$ until the fixed term becomes negligible. When $c_{\text{fixed}}$ is small, as with a function call into an in-process API, batching gains little and costs a great deal in latency, which is why the technique is applied to the wrong layer so often. The first question about any proposed batch is therefore the ratio of fixed to variable cost at the boundary being crossed.

Boundary	Approximate fixed cost	Typical gain from batching
Function call in-process	a few nanoseconds	negligible
Uncontended mutex acquisition	tens of nanoseconds	small
System call	hundreds of nanoseconds	large
Context switch to another thread	microseconds	very large
Network round trip, same datacenter	hundreds of microseconds	dominant
Network round trip, cross-region	tens of milliseconds	everything

The table is about orders of magnitude, and the ordering is what matters. Batching through a network round trip turns a hundred round trips per hundred items into one, which is a hundredfold reduction in the dominant term. Batching through a function call turns nothing into nothing.

Where the delay comes from

A batch has to be assembled before it can be sent, and the assembly takes time. There are two distinct strategies with different delay profiles.

Time-based batching waits a fixed interval and sends whatever accumulated. A batch window of 5 milliseconds means an item that arrives just after a flush waits nearly 5 milliseconds; an item arriving just before the flush waits almost nothing. Mean added delay is half the window, and the distribution over the interval is uniform. This is the strategy used by most aggregating systems because it is simple and bounds memory: at a known arrival rate, the maximum batch is bounded by the window times the rate.

Size-based batching flushes when the batch reaches a threshold. This bounds the delay at the low-rate end trivially (nothing is sent until the threshold is reached, so at low load the delay is unbounded) and is only acceptable when combined with a timeout. In practice both are used together: flush on whichever of count or time comes first. The effective batch size under a combined policy is the minimum of the count threshold and the arrival rate times the window, which is a clean way to see that the tuning is mostly about the arrival rate rather than the code.

Opportunistic batching collects work until the current batch must be committed for correctness reasons (the transaction ends, the lock is released, the buffer must be visible) and thereby pays no added delay at all, because the natural commit points already exist. This is the cheapest form and the one easiest to miss. A function that takes a lock and writes one counter becomes a batched writer for free if the caller already holds the lock and writes several counters.

The tension is that the first two add latency in exchange for throughput, and the exchange rate is not linear. Doubling the batch window doubles the maximum added delay and roughly halves the per-item overhead, but the rate of improvement flattens once the fixed term is small relative to the variable term. A window that reduces per-item overhead from 500 microseconds to 250 is worth a lot; the same doubling from 2 microseconds to 1 is worth almost nothing and costs the same delay.

Batching converts many small transactions into one larger one

The amortization argument extends past call overhead into the behaviour of the layer that receives the batch. A filesystem receiving one large write can allocate contiguous blocks, issue one I/O request per device queue slot, and mark one range dirty in the page cache. A filesystem receiving a thousand small writes may do the same work a thousand times, including a thousand metadata updates that each require a journal record.

The same holds for network stacks, where a batch of small messages can be merged into one segment up to the maximum segment size; for database clients, where a multi-row insert is one statement and one round trip rather than one of each per row; and for write-ahead logs, where a single fsync that covers a thousand records replaces a thousand fsyncs. Each of these is an instance of the same principle: the receiving layer’s per-operation overhead is paid once per batch.

/* Copy items through a buffer, flushing when it fills. */
struct batch {
    char  buf[BUFSZ];
    size_t len;
    int   fd;
};

int batch_push(struct batch *b, const void *item, size_t n) {
    if (b->len + n > sizeof b->buf && batch_flush(b) < 0)
        return -1;                       /* flush failed; propagate */
    memcpy(b->buf + b->len, item, n);
    b->len += n;
    return 0;
}

int batch_flush(struct batch *b) {
    size_t off = 0;
    while (off < b->len) {
        ssize_t w = write(b->fd, b->buf + off, b->len - off);
        if (w < 0) return -1;            /* partial writes are normal */
        off += (size_t)w;
    }
    b->len = 0;
    return 0;
}

The loop around write is not defensive padding. A write to a pipe or a socket may transfer fewer bytes than requested, and a batching layer that assumes one call per buffer loses data on exactly the workloads it was built to speed up.

Failure modes

Batching has failure modes of its own, and they tend to be worse than the inefficiency it removes.

Loss on crash. A batch held in memory is not durable. A process crash loses everything since the last flush; a machine crash loses everything the kernel had not yet persisted. The batch window is therefore a durability window, and a 100-millisecond batch is a promise that up to 100 milliseconds of acknowledged work can vanish. Anything with a durability requirement needs an explicit flush before the acknowledgement is sent, which removes most of the batching gain at the same boundary.

Head-of-line blocking. A large batch takes longer to process than a small one. If one item in the batch requires a retry or is unusually large, every other item behind it waits. Where the batch is transported as a unit with a single acknowledgement, a failed transmission of a batch of a thousand items resends a thousand items, and the cost of the retry grows with the batch size while the probability of any given transmission succeeding does not.

Tail latency amplification. A batch window sets a floor on latency that is independent of load. A service that normally responds in 2 milliseconds acquires a 5-millisecond floor the moment it batches. Nothing in the average response time shows this if the batch amortizes well, but every request now has a fixed component that the clients see as jitter.

Memory under burst. If the arrival rate rises while the batch window is fixed, the batch size rises with it. A window chosen for a steady state becomes a multi-megabyte allocation during a spike, and the allocation itself becomes a cost. Size-based limits in addition to time-based limits bound this.

Reordering and fairness. When items from different sources share a batch, the batch defines a common flush point for all of them. A slow source delays a fast one; a fast source fills the batch and delays a slow one. Batching across independent request streams couples them in a way single-item processing does not.

Choosing where to batch

The right place to batch is the boundary with the highest fixed cost relative to variable cost, which is nearly always the widest boundary crossed. Batching at the application’s own function boundary and then issuing one syscall per item is wasted effort; batching across the syscall boundary by hoisting the loop is worth far more.

There is a second criterion, less obvious: batch only where the items are interchangeable. Items that must be acknowledged individually, ordered relative to each other, or retried independently are poor batch candidates because the batch couples their fates. Items that are contributions to an aggregate, positions in a sequence that must be delivered in order anyway, or records destined for the same file are natural batches.

The limits of the technique appear when the batch itself becomes contended. A single batching writer is a serialization point by construction, and the flushing cost does not disappear, it moves. How far that serialization can be pushed is the subject of Sharded Counters, Padding, and False Sharing; how the batch’s arrival rate is controlled from upstream is the subject of Bounded Queues and Backpressure as Throughput Control.