Durable Buffers and the Cost of At-Least-Once Export

An in-memory export queue survives a backend outage as long as the process stays alive. It does not survive a restart, a crash, a deployment, or the out-of-memory killer. Those four events occur most often during incidents, which is exactly when the telemetry is most valuable. A durable queue closes that gap by placing records on disk before acknowledging them, and the price is a write and a flush on the path of every record.

What durability actually requires

A record is durable when it survives an abrupt loss of power. Anything weaker, surviving a process crash but not a machine failure, or surviving a machine failure but not a disk failure, is a different guarantee with a different cost, and the pipeline’s configuration usually does not say which one it provides.

The chain of buffering between an application write and the physical medium is long: the process’s memory, the kernel’s page cache, the storage device’s write cache, and the medium itself. A write returns as soon as the data is in the page cache, so a process crash does not lose it but a machine failure does. A flush and sync pushes it to the device, but a device with a volatile write cache acknowledges before the data is on the medium unless a cache-flush command is issued. The sync operation of the file API covers the first step; the cache flush covers the second, and whether the kernel issues it depends on the filesystem and on the device’s declared cache behaviour.

The relevant cost is therefore a round trip to a device rather than a memory copy. On a local solid-state device the round trip is on the order of tens of microseconds when no other operation is in flight; on a network-attached volume it is a network round trip, which is two to three orders of magnitude larger. The number that matters for a pipeline is not the latency of one operation but the number of operations that can be outstanding concurrently, because a queue that serializes one flush per record is limited to a rate determined by that round trip.

The segment log and the acknowledgement boundary

A durable queue is almost always a log: an append-only sequence of segments, each a file, with a position that advances monotonically. Appends go to the active segment; when it reaches a size bound, a new segment is created and the old one is closed. Consumption is tracked by a cursor per consumer, and a segment is deleted once every consumer’s cursor has passed its end.

The ordering inside the log is the ordering of enqueue, which gives the queue a property its in-memory predecessor lacks: replay after a restart produces records in the order they were first seen. For telemetry that ordering is not semantically required, records carry their own timestamps, but it makes reasoning about the queue’s contents simple, and it permits the consumer to resume from a stored position rather than from an arbitrary point.

The acknowledgement boundary is the design decision that sets the queue’s cost. Two boundaries are possible.

Acknowledge on write. The producer’s append returns, and the record is considered accepted, as soon as the bytes are in the file, buffered in the page cache, not necessarily on the medium. Throughput is high because many appends share one flush. Durability is bounded by the page cache’s writeback interval, so a machine failure can lose the last interval’s records.

Acknowledge on sync. The append returns only after a sync has confirmed the data is on the medium. Durability is real, and the producer’s rate is capped by the sync rate.

A third, common arrangement groups the sync: several concurrent producers append, one of them takes a lock and issues a single sync covering all their writes, and all of them are acknowledged when it completes. This is group commit, and it converts a per-record sync cost into a per-group cost without weakening the guarantee. Its throughput is bounded by the sync round trip divided by the group size, and the group size is bounded by the number of producers willing to wait.

/* Append and group-commit. Returns only after the record is on the medium. */
int log_append(struct log *l, const void *buf, size_t len) {
    pthread_mutex_lock(&l->mu);
    if (l->active_len + len > l->segment_max) {
        if (log_roll(l) != 0) { pthread_mutex_unlock(&l->mu); return -1; }
    }
    ssize_t n = pwrite(l->fd, buf, len, l->active_off);
    if (n != (ssize_t)len) { pthread_mutex_unlock(&l->mu); return -1; }
    l->active_off += len;
    unsigned long seq = ++l->pending;

    /* One sync covers every append already written; waiters share the result. */
    if (l->sync_in_flight) {
        pthread_cond_wait(&l->synced, &l->mu);
        int ok = (l->synced_seq >= seq);
        pthread_mutex_unlock(&l->mu);
        return ok ? 0 : -1;
    }
    l->sync_in_flight = 1;
    int rc = fdatasync(l->fd);
    l->synced_seq = seq;
    l->sync_in_flight = 0;
    pthread_cond_broadcast(&l->synced);
    pthread_mutex_unlock(&l->mu);
    return rc;
}

The fdatasync call is the part that cannot be optimized away: without it, the operating system’s promise about the file’s content stops at the page cache. A pipeline that claims durability without a sync is claiming something the kernel does not provide.

Deletion is the operation that fails under load

Enqueue and dequeue are the operations that get tuned, and deletion is the one that breaks. A segment becomes eligible for removal only when every consumer’s cursor has passed it, so a single slow consumer holds the entire log behind it. The disk fills, new appends fail, and the producer’s append returns an error.

The failure mode is a coupling between consumers that the topology did not intend: one consumer that stops reading, because its backend is down, stops the durable queue from reclaiming space, and the producer eventually has no space to write. The producer’s only options are to block, which propagates the slow consumer to the source, or to drop, which discards the oldest records.

Bounding the log by size rather than by consumer position is the standard resolution: a maximum disk budget and an eviction policy that drops the oldest unread segment when the budget is exhausted. The eviction policy converts a hard failure (no space, append error) into a graceful one (records lost from the oldest end, counted). Which is preferable depends on what the queue is for, a compliance archive tolerates a blocked producer and not a lost record, and an operational trace pipeline tolerates a lost record and not a stalled application. Shedding at the producer, described in Load Shedding and Retry Amplification, is the same choice made one hop earlier and on richer information.

Retention by time is a third bound, and it is the one that matters when a consumer is offline for longer than any record remains interesting. A log that keeps a fixed number of hours loses the difference between “the consumer is slow” and “the consumer is gone”, in both cases the data ages out.

At-least-once and the operations it does not support

The guarantee is that every record that was accepted is delivered at least once. The complementary statement, that a record is never delivered twice, is not offered, and the reason is structural: the acknowledgement of delivery happens after the delivery, and a failure in between forces the sender to choose between retransmitting and losing the record. At-least-once is that choice made in one direction.

The consequence is that every consumer must be tolerant of repetition. That tolerance is not uniform across data types:

Counters tolerate repeats if the storage layer deduplicates by series identity and timestamp, since two samples with the same timestamp and series are the same observation. Without deduplication, a repeated counter sample read twice inflates a rate computed by differencing.

Spans tolerate repeats if the backend keeps the last write for a given span identifier, since both copies describe the same span. Two copies that differ, a status set after the fact, resolve to the later one only if the backend compares versions rather than accepting the first arrival.

Log lines do not tolerate repeats at all, because a repeated line is indistinguishable from a genuine repeated event. A log store cannot deduplicate without a unique identifier per line, and most log formats do not carry one.

That asymmetry is the reason a single durable queue serving several signal types needs per-signal deduplication policies rather than one. It also explains why the duplicate analysis in Fan-Out Topologies and Duplicate Export is not only about topology: the same duplication is benign for one signal and corrupting for another.

Ordering across a restart

A single log preserves order across a restart because the cursor is stored with the data. A pipeline with several queue stages does not: a record passes through a disk queue at the agent, a network hop, and a second disk queue at the collector, and the restart of any stage reorders records relative to those that were already in flight downstream.

For traces the reordering is harmless because span start times are in the spans. For metrics it matters, because a counter sample that arrives after a newer sample from the same series is a decrease and is interpreted as a reset. A backend that sorts samples by timestamp before differencing is immune; one that applies them in arrival order is not.

The other reordering source is the retry itself. A batch that fails and is retried arrives after batches that succeeded later, so a successful batch’s samples can precede an earlier batch’s samples at the receiver. Again, the timestamps in the records are the only reliable ordering, and any pipeline component that assumes arrival order is assuming something the durability mechanism explicitly does not provide.

Where a durable queue is worth it and where it is not

The queue costs a write to disk per record, plus a sync if durability is claimed, plus the storage for retention. Its benefit is that a restart does not lose the buffer, typically the last few seconds of telemetry, proportional to the batch interval.

For a high-volume trace pipeline with head sampling at a low rate, the loss on restart is already small, and the queue’s cost is a disk write for every retained span. For a log pipeline carrying audit records, the loss is unacceptable and the queue is mandatory. The decision is the ratio of the value of the last batch before a crash to the cost of writing every batch, and that ratio differs by signal by orders of magnitude.

A middle option exists and is worth naming: a bounded in-memory queue with a spill to disk only when the queue is under pressure. The steady state costs nothing on the write path, and the failure mode, a restart while the queue is small, loses only what was in memory. The complexity is that the spill path is rarely exercised, so it is rarely correct, which is the classic argument against the hybrid and the reason most deployments choose one of the two pure designs.