Load Shedding and Retry Amplification

A telemetry pipeline is an amplifier. Its input is generated by every process in the fleet, and the rate of generation is a property of the workload rather than of the pipeline. When the workload grows, or an incident causes every service to log and trace more heavily, the pipeline receives more than it can process, and the only question is what it does with the excess. Doing nothing is a choice: it means the excess accumulates in a queue until memory or disk runs out, and the failure arrives all at once.

The arrival rate is not under the pipeline’s control

Load shedding is the deliberate discard of work that cannot be served, made at a point where the discard is cheap. Its precondition is that the pipeline knows it is overloaded, and that knowledge must be local and fast. A pipeline that discovers overload by watching its own latency has already accumulated a queue proportional to the excess times the detection delay.

The signals available to a receiver are few and immediate:

Queue occupancy. The depth of the internal queue as a fraction of its maximum. This is the most direct signal, and it is also the one that must be interpreted with care, because a queue that is full because of a batch flush policy is not overloaded, it is working. Occupancy sustained above a threshold across several flush intervals is a reliable signal; an instantaneous reading is not.

Concurrency saturation. The number of in-flight requests against a fixed limit. When every slot is occupied and the wait for a slot is non-zero, the receiver is at capacity regardless of what its CPU metrics say.

Per-request latency against a budget. If the time to process a batch exceeds the interval at which batches arrive, the queue grows without bound even though each individual request was served.

The useful signal is a combination, and the common mistake is to use CPU utilization alone. A pipeline that is CPU-bound and one that is blocked on disk I/O in a durable queue have very different shedding responses, in the first case the shed reduces work and helps; in the second it reduces work that was already committed to disk and helps less.

Shedding at the edge rather than at the end

The cost of discarding a record depends on where it is discarded. At the receiver’s entry point it costs nothing beyond the parse that identified the record. Inside a tail sampler’s buffer it costs the memory it occupied and the work already spent assembling its trace. In a durable queue it costs a disk write and a later deletion.

The consequence is that shedding belongs as early as possible, which means at the agent rather than the collector, and at the collector’s receiver rather than in its processors. The agent knows it is producing more than the network can carry because its export queue is full; that is the moment to discard. Discarding there is also more effective, because the records never traverse the network, so the shed relieves both the sender and the receiver.

An agent that sheds needs a policy about what to shed. Three policies cover the useful cases.

Drop newest. The incoming record is discarded and the buffer retains its contents. The retained window is the period before the pressure began, which is often exactly the period an investigator cares about, the onset of an incident precedes the peak.

Drop oldest. The buffer is rotated so that recent records survive. This favours live debugging and discards the history that explains how the current state arose.

Drop by class. Records are tagged with a priority at their source, and the shed discards low-priority classes first. Errors and warnings survive while debug and info are discarded. This requires the producer to set the priority, which means it requires a convention across the instrumentation, and it is the only policy that preserves the signal most useful during an incident.

Priority classes are the most valuable of the three and the most often omitted, because they require a change at the source rather than a configuration change in the pipeline. The class should be a small enumeration, not a numeric score, since scores invite debate about the exact value and a small enumeration forces a decision.

Retry amplification, precisely

A retry is an attempt to convert a transient failure into a success. When the failure is caused by load, the retry adds load, and the mechanism that was supposed to help extends the outage.

The arithmetic is straightforward. Suppose a backend serving n senders fails requests with probability f under current load, and each failed request is retried up to k times. The offered load becomes n * (1 + f + f^2 + ... + f^k) in the worst case, which for f = 1 is n * (k + 1). A backend that is merely saturated, and therefore failing a large fraction of requests, is subjected to several times its capacity by the retries, and its failure probability rises as a result. The system has positive feedback, and it reaches an equilibrium at total failure rather than at degraded service.

Three mechanisms break the loop.

A retry budget caps the retried fraction. When the budget is exhausted, failures are propagated as failures rather than retried. The total load is bounded at n * (1 + b) where b is the budget fraction, regardless of how many failures occur. A budget expressed as a ratio rather than a count adapts automatically to the offered load.

Jittered exponential backoff spreads retries in time. Without jitter, senders that failed at the same instant retry at the same instant, producing a synchronized wave whose amplitude is the sum of all senders’ retry rates. With full jitter, a delay drawn uniformly from zero to the exponential bound, the retries are spread across the interval and the peak load is n rather than n * k.

A server-directed delay. A backend that is overloaded can tell senders to wait, and well-behaved senders honour it. An explicit retry-after indication in a response, or a “too many requests” status, conveys both the fact of overload and a suggested delay. This is the only mechanism in which the overloaded component participates in its own relief, and it is why the response codes matter more than the client’s scheduling logic.

The amplification that servers cause

The client side is only half of the picture. A load balancer or proxy in front of a collector can amplify load on its own, in three ways that are easy to configure accidentally.

Automatic retry on upstream failure. A proxy that retries a request when the upstream connection breaks will resend a request the upstream may have already processed. During a partial failure, one backend in a pool dying, this doubles the load on the remaining backends exactly when they are absorbing the dead one’s share.

Health-check-driven removal. A backend that is slow because it is loaded fails its health check, is removed from the pool, and shifts its load to the others, which then become slow and fail theirs. The pool collapses to empty. The remedy is a health check that distinguishes “process is alive” from “process is keeping up”, and a load balancer that keeps a slow backend in the pool while directing a reduced share to it.

Connection limits interpreted as failures. When a backend’s listen backlog fills, new connections are refused; the refusal looks identical to the backend being down, and senders respond by retrying more aggressively. A backend that is merely at its connection limit receives the same treatment as one that has crashed.

Each mechanism is individually reasonable and collectively catastrophic. The design rule is that only one layer should retry, and that layer should have a budget. A proxy without retries in front of clients with budgets behaves predictably; a proxy with retries in front of clients with retries does not.

Shedding downstream, and the shape of the loss

A collector that has shed at its edge and still cannot keep up must shed downstream as well, and the choice of what to drop determines the shape of the data that survives.

Dropping whole batches preserves the integrity of traces and the continuity of time series within a batch, at the cost of gaps. Dropping individual records preserves coverage across time while breaking the completeness of traces, a trace with a dropped span is a trace with a hole, which is worse than an absent trace for causal analysis.

Shedding by tenant or by service protects the pipeline’s ability to serve its most important consumers, at the cost of silencing others completely. Shedding by attribute value is the same policy expressed more finely: drop the noisy tenant’s debug logs and keep everything else. Either form depends on the attribute being present at the moment of the decision, which is a constraint on where the shed sits relative to the enrichment stages, a rule keyed on a tenant attribute cannot run before the stage that attaches it, and the ordering consequences of that are set out in Enrichment Order and Redaction in Flight.

The choice must be documented, because the shape of the loss determines what a consumer can conclude. A gap in a time series is not evidence that the metric was zero; an absent trace is not evidence that the request failed. The strongest available mitigation is to emit the loss itself as a metric, a counter of dropped records by reason and by class, so that the gap has an annotation and a query can distinguish “no data because nothing happened” from “no data because it was discarded”.

The configuration surface that matters when it is failing

Four settings determine whether a pipeline degrades gracefully or collapses, and they are worth reviewing before an incident rather than during one.

The queue size in memory and on disk, which bounds the buffering and therefore the delay before failure.

The failure policy for each exporter: retry with a budget, drop, or block. Blocking on a telemetry path is a decision that a collector’s unavailability stops the application, and it should be made explicitly.

The concurrency limit per receiver, which is the point at which the receiver starts refusing rather than queuing.

The shedding policy itself, with its priority classes and its accounting.

A pipeline with all four specified fails in a predictable direction: it loses the least valuable records first, it reports the loss, and it recovers as soon as the overload passes. A pipeline with none of them specified does not fail at all in the ordinary sense, it holds the excess until memory is exhausted, and the failure it produces is a kill signal that takes down every signal the process was carrying, including the ones that were far from the bottleneck.

The relationship between shedding and the earlier stages of a pipeline is worth stating once. Sampling reduces volume before the pipeline, batching and compression reduce cost per record, and shedding decides what to do when the reduction was insufficient. A pipeline configured with generous sampling and no shedding is a pipeline that has chosen to fail hard under load; one configured with shedding and no sampling is one that discards under pressure what it could have discarded deliberately. The two must be designed together, and the buffer bounds in Durable Buffers and the Cost of At-Least-Once Export are the bridge between them.