Batching and Flush Policy in Telemetry Agents

A process that exports each telemetry record the moment it is produced spends most of its network time on things that are not the record. A single span of a few hundred bytes travels inside a transport frame, a request header block, and a TLS record, and arrives at a collector that must decode and acknowledge a message whose envelope is comparable in size to its contents. Batching exists to pay those costs once per group instead of once per record. The design question is not whether to batch but how the batch boundary is chosen, because the choice sets both the memory ceiling and the worst-case delay of every record in the stream.

What a batch amortizes, and what it does not

Three distinct costs shrink when records are grouped.

The transport cost is per message: connection setup, request headers, HTTP/2 frame headers, gRPC message framing, and the acknowledgement round trip. With HTTP/2, a header block for a small request is often larger than the payload it describes, and the header compression state is per connection, so reusing one long-lived connection for many batches matters as much as the batch size itself. Sending one record per request means the cost per record is dominated by a constant that does not appear in the record at all.

The encoding cost is partly per message. Protocol Buffers encodes repeated fields with a tag per element, but a nested message field is written once with a length prefix. OpenTelemetry’s data model puts resource attributes in a Resource that a batch of spans shares, so an agent shipping 500 spans in one request writes the service name, host name, and every other resource attribute once rather than 500 times. On a service with a dozen resource attributes of 20 bytes each, that is roughly 240 bytes per span that disappear from the wire.

The compression cost is per byte, but compression ratio improves with input size because the compressor’s window has more history to match against. A 200-byte payload compressed with gzip frequently grows; a 200-kilobyte payload typically shrinks by a large factor. Batching and compression are therefore multiplicative rather than additive, which is why the two are almost always configured together, the accounting is worked through in Compression and the Real Cost of an Ingested Byte.

What batching does not amortize: per-record timestamps, identifiers, attribute keys, and any field whose value differs between records. Those scale linearly no matter how records are grouped, and they are the reason cardinality control is a separate problem from batching.

Two triggers, one effective latency

A batch accumulator is almost always driven by two conditions, whichever fires first:

• a size trigger, fired when the accumulated record count or encoded byte count reaches a threshold;
• a time trigger, fired when the oldest record in the buffer has waited longer than an interval.

The size trigger protects memory and keeps individual requests below transport limits. The time trigger protects latency. Their interaction is simple and frequently misread: the delay experienced by a record is bounded by the interval, not by the size. If traffic is heavy, the size trigger fires first and records leave sooner than the interval. If traffic is light, the interval dominates, and the last record in a batch waits almost the full interval while the first waited almost none.

For arrivals spread evenly across the interval, the mean wait is about half the interval, and that is the number to use when reasoning about how fresh the data at the backend is. An interval of ten seconds means the visible lag of a low-traffic signal is on the order of five seconds plus export time, not ten. The distribution is what matters for alerting: a query that evaluates a rate over a one-minute window against data that arrives in ten-second batches sees a step function rather than a smooth curve.

An accumulator that also caps the encoded size must handle the case where one record exceeds the cap. There is no correct generic answer. Splitting a record is impossible without changing its meaning, so the accumulator must either emit the oversized record alone or reject it, and the choice has to be visible in the configuration rather than accidental.

Flushing on an empty buffer

The time trigger needs a timer, and a naive implementation restarts the timer on every append, which converts the interval into an idle timeout: under continuous traffic the batch never flushes on time and grows until the size trigger or the memory limit stops it. The correct trigger measures the age of the oldest buffered record, not the time since the last append. Two implementations get this right in different ways: scheduling a timer when the buffer transitions from empty to non-empty and cancelling it when the buffer drains, or comparing a stored first-enqueue timestamp against the current time on each append.

A timer that fires against an empty buffer should do nothing. If it issues an export request anyway, an idle agent sends a stream of empty messages, each of which costs a round trip and consumes a request slot on the receiving side. Receivers generally accept empty batches, so the failure is invisible except in request counts and in the load the receiver carries.

There is a second, subtler consequence of the timer: batches produced by the time trigger and batches produced by the size trigger interleave, and if exports run concurrently, two batches can reach the backend out of order. Spans carry their own timestamps, so the ordering of records within a trace is recoverable, but the arrival order at an ingest endpoint is not preserved and any backend logic that assumes monotonic arrival, deduplication by last-write-wins on a mutable record, for instance, will behave incorrectly under concurrent exports to the same destination.

Sizing against the transport

Batch size ceilings are usually set by what the transport will accept rather than by what the backend prefers. gRPC implementations commonly default to a 4 MiB maximum received message, so a batch encoded near that limit is accepted only if both sides agree, and a batch slightly over it is rejected as a whole. HTTP/2 frames are 16 KiB by default, so a multi-megabyte body is split into many frames and the receiver’s flow-control window, not the frame size, governs throughput.

The practical constraint is usually the receiver’s memory multiplied by the number of concurrent senders. A collector configured with a fixed receive buffer serving a thousand agents each willing to send 4 MiB has a worst case that is not survivable. Agents are therefore configured far below the transport limit, tens to hundreds of kilobytes per batch is a common range, and the collector’s own limits are set independently.

Backpressure, drops, and the queue between them

Batching implies a queue, because the producer runs at application speed and the exporter runs at network speed. The queue’s policy under overflow is the most consequential setting in the whole agent, and it has exactly three honest options.

Block the producer. The append call waits until space is available. This propagates the backend’s slowness into the application, which is correct for a signal the application cannot proceed without and usually wrong for telemetry: a logging call on a request path that blocks on a remote collector turns a collector outage into an application outage.

Drop. The append call discards a record when the queue is full, either the newest or the oldest. Dropping the oldest keeps the most recent window, which is what an operator debugging a live incident wants; dropping the newest keeps a contiguous prefix and is easier to reason about. Either way the drop should be counted, because an uncounted drop is indistinguishable from a signal that never existed.

Grow without bound. This is not backpressure; it is deferred failure. An unbounded queue under sustained overload converts a throughput problem into an out-of-memory kill, and it does so after a delay long enough that the cause is no longer visible in the process’s own metrics.

type Batcher struct {
	mu      sync.Mutex
	buf     []Record
	maxLen  int
	maxAge  time.Duration
	firstAt time.Time
	out     chan<- []Record
}

// Add returns false when the buffer is full and the record is dropped.
func (b *Batcher) Add(r Record) bool {
	b.mu.Lock()
	defer b.mu.Unlock()

	if len(b.buf) == 0 {
		b.firstAt = time.Now()
	}
	if len(b.buf) >= b.maxLen {
		return false // caller counts the drop; the queue stays bounded
	}
	b.buf = append(b.buf, r)
	return true
}

// Tick is driven by a caller-owned ticker. It flushes on age, never on idleness.
func (b *Batcher) Tick(now time.Time) {
	b.mu.Lock()
	if len(b.buf) == 0 || now.Sub(b.firstAt) < b.maxAge {
		b.mu.Unlock()
		return
	}
	batch := b.buf
	b.buf = nil
	b.mu.Unlock()

	select {
	case b.out <- batch:
	default:
		// export queue full: the batch is dropped, not buffered again
	}
}

The default branch in the final select is the drop decision made explicit. The alternative, blocking until the exporter accepts the batch, moves the stall back into Tick, and from there into whatever goroutine drives it.

What the last interval costs at shutdown

A process that receives a termination signal and exits without flushing loses everything in the buffer, which under a ten-second interval is up to ten seconds of telemetry from every instrumented process at once. This is precisely the window in which a crash or a deployment is most interesting. Agents therefore install a shutdown path that stops accepting new records, flushes the buffer once with a bounded timeout, and then exits; the timeout is required because a backend that is unreachable must not prevent the process from terminating.

The same reasoning applies to a collector restart during a rolling upgrade: without a flush and a durable queue, the batch in memory is lost on every restart, and the loss is periodic rather than accidental. Durable queues address that, at the cost of disk I/O on the write path, as described in Durable Buffers and the Cost of At-Least-Once Export.

Batching is the cheapest lever in a pipeline: it reduces bytes, request count, and receiver CPU simultaneously. It is also the one that most directly trades against latency, which is why the interval belongs in the same discussion as the sampling and delivery decisions: a sampled-out record costs nothing, a batched record costs less, and a batch that is dropped on shutdown costs everything that was in it.