Backpressure and Loss at the Collection Boundary

The logging call and the log store are separated by a path with finite capacity: a pipe, a socket, a collector’s memory, a remote ingestion endpoint. When the store falls behind, the path fills, and something must give. Three options exist at the boundary: block the producer, drop records, or buffer without limit. The third is not really an option; it is a delayed decision between the other two, made by the out-of-memory killer.

The three regimes and who pays

The producer’s write to a pipe or socket is the point where the decision is enforced, because that call has a definite behaviour when the buffer is full.

A blocking write parks the calling thread until space is available. If the logging call is on the request path, the request now waits for the log store. A log store that is slow because it is overloaded now slows the application, and the application’s slowdown increases the time each request holds its resources, which increases concurrency, which increases load. This is a feedback loop with a positive gain, and it is the mechanism by which a logging outage becomes an availability outage. The loop is real and has been described as a general pattern in queueing terms: a service whose capacity is reduced by a dependency’s latency degrades faster than the dependency.

A non-blocking write returns an error such as EAGAIN when the buffer is full. The producer must then drop the record or retry later. Dropping is immediate and bounded; retrying later requires a buffer in the producer, which moves the problem one level up. The O_NONBLOCK flag is what makes this regime possible, and it is a per-file-descriptor property rather than a per-call one, so the choice is made when the descriptor is opened.

A bounded in-memory queue decouples the producer from the writer, at the cost of a delay and a fixed capacity. The delay is the queue depth divided by the drain rate, which is small when the store is healthy and unbounded when it is not. The capacity is the only thing that bounds memory, and it is the number that decides when drops begin.

The regime is chosen per log path, and it should be chosen per record class. An audit record and a debug trace do not belong in the same queue with the same policy, because losing one is a compliance problem and losing the other is normal.

Sizing a buffer with arithmetic

Buffer sizing is a product of rate and tolerance: how long an outage the system must survive, multiplied by the rate at which records accumulate during it.

A producer emitting 5,000 records per second at 400 bytes each generates about two megabytes per second. Surviving a five-minute ingestion outage requires a buffer of roughly 600 megabytes, which is not a figure most services can reserve. Surviving thirty seconds requires 60 megabytes, which is plausible. The arithmetic is unforgiving, and it is the reason local disk queues exist: a queue on disk costs storage rather than memory, so it can be sized in gigabytes without competing with the application’s heap.

The disk queue introduces its own costs. Every record is written twice, once to the queue and once to the store, so the write amplification is at least two, and the queue’s write path must be durable enough to survive the process restart it exists to survive, which means an fsync policy. A queue that buffers writes in the page cache and loses them on a power failure is a queue that fails exactly when it is needed most.

The second cost is the replay rate. After an outage, the queue holds a backlog and the shipper must drain it. If the drain is unbounded, the store receives a burst that may be larger than it can handle, reproducing the original overload. The drain rate therefore needs its own limit, and the backlog drains over a period longer than the outage. This is the same shape as TCP’s behaviour after a loss event: the sender does not resume at the pre-loss rate, because the path has not been shown to support it.

Flow control that pushes back in the right place

The direction of backpressure matters more than its existence. In a pipeline of application, local shipper, collector, and store, a limit applied at the wrong hop either does nothing or amplifies.

If the store applies backpressure to the collector, and the collector applies it to the shipper, and the shipper applies it to the application, the limit reaches the producer and the application slows. If any hop in that chain buffers without bound, the limit does not propagate and the hop absorbs the pressure until it exhausts memory. The design question is where the propagation stops.

The usual answer is that propagation stops at the local shipper, which is the component whose loss is tolerable and whose buffer is on disk. The application writes to a local pipe or a local socket with a bounded buffer; if the shipper cannot keep up, the pipe fills, and the application’s logging call blocks or drops according to its policy. This keeps the failure local to one host and prevents a fleet-wide logging outage from becoming a fleet-wide application outage, at the cost of losing records on that host.

A subtler variant separates the classes: severe records take a blocking path and debug records take a dropping path, so that under pressure the stream degrades to the records that matter. That requires the level check to happen before the queue, which is the same ordering requirement as Severity Levels as a Filtering Contract. A level filter applied after the queue saves storage but not the queue slot, and the queue is where the pressure is.

Accounting for what was dropped

A system that drops records must count them, and the count must be visible in the same place as the records it failed to deliver. A drop that is invisible is worse than a drop that is loud, because a gap in a log stream is indistinguishable from an absence of events. The operator’s inference “there are no errors after 14:02” is wrong, and nothing in the data contradicts it.

The minimum accounting is a per-reason counter, incremented at the point of loss and exported as a metric. Reasons differ and matter: a queue-full drop during an outage is expected and is recovered by draining; a record-too-large drop is permanent and indicates a producer bug; a parse failure at the collector is permanent and indicates a format mismatch. Aggregating them into one counter hides the distinction.

The counter must also survive the process. A shipper that keeps its drop count in memory loses the count when it restarts, which is precisely when drops are most likely. Periodic persistence, or emitting the count as a record of its own, closes the gap.

There is a second-order problem: the drop counter is itself a signal that flows through the same path. If the path is down, the counter cannot be delivered, so the count arrives late or not at all. A local exposure, such as a file or an endpoint the monitoring system scrapes separately, decouples the accounting from the delivery path. That separation is the difference between knowing that records were lost and inferring it later.

Degenerate cases

Three cases defeat a straightforward implementation.

A single huge record. A record larger than the queue’s slot size cannot be enqueued at all. The failure appears as a permanent drop with a distinct reason, and the fix is at the producer: the size policy described in Record Framing: Why a Log Line Needs a Delimiter Contract belongs at the call site, not at the shipper. A shipper that truncates silently converts a visible problem into an invisible one.

A blocked reader with a full pipe. If the reader of a pipe dies and the writer does not handle EPIPE or SIGPIPE, the writer receives a signal that by default terminates the process. A logging path that can kill its own process is a liability, and the signal must be ignored or blocked in the logging thread. The alternative, a write to a closed socket returning an error, is the behaviour to prefer, and it requires a check that the code may not perform if the write is inside a library.

Sustained overload at a rate below the drop threshold. If the store is slightly slower than the producer, the queue fills slowly, and the system sits at the boundary for hours. Records are dropped intermittently, the queue depth oscillates, and the visible symptom is an inconsistent stream rather than a clean outage. Queue depth is the metric that reveals this regime; a depth that stays pinned at capacity is a queue that is dropping continuously.

The compression that reduces the bytes flowing through this path is covered in Block Compression and Columnar Layout for Log Storage, and the retention window that determines how long a drained backlog remains useful is covered in Retention Policy as a Storage Budget.