Adaptive Sampling: Feedback Control over Span Volume

A fixed sampling rate is a bet placed once. Configure one percent and the volume is predictable only if the traffic is; a service that normally handles a hundred requests per second and peaks at five thousand during a promotion produces fifty times the span volume during exactly the period when the tracing backend is least able to absorb it. The alternative is to make the rate an output of a control loop: measure the rate of spans produced, compare it to a target, and move the sampling probability to close the gap.

The idea is simple. The execution is where the difficulty lives, because a rate that changes is a rate that corrupts the statistics computed from it, and a control loop that reacts to its own output can oscillate. This is a refinement of the fixed-rate approach described in head-based sampling, and it inherits that approach’s blindness to the outcome: whatever the controller does, the decision still precedes the work.

The loop and its gain

The controller has one measurement, one target, and one actuator. The measurement is spans per second emitted by the service, or the trace decisions per second, whichever the budget is expressed in. The target is a number chosen to fit within the backend’s ingest capacity, divided among services. The actuator is the sampling probability, which for probabilistic sampling is directly proportional to the emitted volume.

A naive proportional controller adjusts the probability by the ratio of target to observed, once per interval. That is stable when the measurement reflects the new probability, but it does not, because spans emitted during an interval were decided at their start using the previous probability. The measurement lags the actuator by one interval, and a controller with too much gain on a lagged measurement oscillates: it cuts the rate, observes low volume, raises the rate, observes high volume, and repeats. The oscillation is worst at high traffic, because the absolute change per step is largest there.

The practical fixes are standard control engineering. Damping, by moving the probability only a fraction of the way toward the computed value, slows the response and removes most of the oscillation. A deadband, in which no adjustment happens while the measurement is within some percentage of the target, prevents the loop from reacting to ordinary variance, which for sampled counts is large at low volume. And a clamp on the per-step change bounds the worst case.

An exponentially weighted moving average of the measurement is the usual smoothing. The weight determines the effective window: a weight of 0.2 means the average is dominated by roughly the last five intervals, which at a ten-second interval is under a minute of history. A weight near zero is smooth and slow; a weight near one is noisy and fast.

type AdaptiveSampler struct {
	mu        sync.Mutex
	rate      float64 // current probability, in (0, 1]
	target    float64 // desired traces per second
	observed  float64 // EWMA of traces per second
	alpha     float64 // EWMA weight, e.g. 0.2
	maxStep   float64 // largest multiplicative change per interval
}

// Adjust runs once per interval with the raw count of decisions made.
func (s *AdaptiveSampler) Adjust(decisionsInInterval float64, seconds float64) {
	s.mu.Lock()
	defer s.mu.Unlock()

	instant := decisionsInInterval / seconds
	if s.observed == 0 {
		s.observed = instant
	} else {
		s.observed = s.alpha*instant + (1-s.alpha)*s.observed
	}

	// Deadband: ignore deviations within 10% of target.
	if s.observed < s.target*0.9 || s.observed > s.target*1.1 {
		desired := s.rate * s.target / s.observed
		// Bound the step so one noisy interval cannot swing the rate.
		lo, hi := s.rate/s.maxStep, s.rate*s.maxStep
		if desired < lo {
			desired = lo
		} else if desired > hi {
			desired = hi
		}
		if desired > 1 {
			desired = 1
		}
		if desired < 1e-6 {
			desired = 1e-6
		}
		s.rate = desired
	}
}

The floor at a very small probability is not cosmetic. Without it, a period of very high traffic drives the rate toward zero, and when the traffic subsides the loop must climb back from almost nothing, which takes many intervals and leaves the service untraced for the duration. A floor guarantees that some traces are always kept.

Why one rate for the whole service is wrong

A service does not have one traffic mix. It has a handful of endpoints or operations with wildly different volumes: a health check at thousands per second, an order submission at tens per second, an administrative export at a few per hour. A single rate tuned to keep the total within budget samples all of them equally, which means the rare operations are effectively never traced and the high-volume ones dominate the sample.

The standard remedy is a per-operation rate, with the budget divided among operations. The division can be equal, which favours the rare operations, or proportional to volume, which preserves overall proportions, or a hybrid in which each operation gets a small guaranteed share and the remainder is divided by volume. The guaranteed share is what makes the rare operation visible at all: without it, its expected trace count is proportional to its share of traffic, which may be one trace per day.

The mechanism for keeping the per-operation decisions consistent is the same as for any head-based decision: the decision is made at the root and propagated in the sampled flag. When the root operation is known at the entry point, the per-operation rate can be selected there. When it is not, as with a generic HTTP handler that dispatches on path, the rate must be selected after routing, which means the decision point moves into the handler and the propagation must still happen before the first downstream call.

A second dimension is the service instance. A rate computed per instance independently means ten instances each aim at the same target and produce ten times the intended volume. The target must be divided by the number of instances, or the controller must run centrally and publish the rate to instances. The first is simpler and fails when the instance count changes; the second requires a coordination channel and fails when the channel is unavailable. Either way, the rate must not be a per-instance independent decision.

The statistics problem

Here is the cost that makes adaptive sampling harder to defend than it first appears.

A count of sampled traces can be scaled to an estimate of total traces by dividing by the sampling probability, but only if the probability is known and constant for the traces being counted. With an adaptive rate, each trace was sampled at a different probability, and the correct estimator is a sum of inverse-probability weights: each kept trace contributes 1/p_i to the total, where p_i is the rate in effect when that trace was decided. Averaging the rates or using the final rate is wrong in a way that is systematically biased, not merely noisy.

The consequence is that the probability must be recorded per trace. The natural place is an attribute on the root span, since the root is where the decision was made, and the value should be the actual probability used rather than the configured target. A trace that survives into storage without its sampling probability cannot be reweighted later, and a dashboard that computes rates from counts will be wrong by whatever factor the rate drifted.

The second statistical cost is that a changing rate breaks time-series comparability. A latency percentile computed over sampled spans is an estimate whose variance depends on the sample size, and the sample size changes with the rate. A period with a low rate has wider confidence intervals than a period with a high rate, and a dashboard that plots both without distinguishing them shows a change in estimate quality as if it were a change in the system. Where the percentile is computed from the sampled data rather than from a separate metrics pipeline, the sampling rate has to be displayed alongside it or the chart is misleading.

Interaction with the rest of the pipeline

An adaptive head sampler and a tail sampler compose in a way that requires care. The tail stage’s input volume is the head stage’s output, so a head controller targeting a fixed span rate keeps the tail stage at a predictable load, which is good. But the tail stage’s retention policies are expressed as rates over its input, and when the head rate drops during a traffic spike, the tail stage sees fewer traces at exactly the moment it wants more error traces. The two controllers can fight: the head loop sees a spike and reduces the rate, the tail loop sees fewer errors and raises its retention, and neither knows about the other.

The fix is a hierarchy rather than two independent loops, with the head stage feeding a stage whose buffering behaviour is described in tail-based sampling: the head rate is set to a level that keeps the total pipeline within capacity, and the tail policies are expressed as absolute budgets with priorities, so that error retention is guaranteed up to a cap and the remaining capacity is filled by the sampled remainder. The head rate can then be allowed to drop during spikes without starving the tail stage of its highest-priority input, because the tail stage’s error retention is bounded by its own cap rather than by the input rate.

One further constraint is the size of the trace itself. A rate that keeps the number of traces constant does not keep the number of spans constant, because trace size varies. A service that starts emitting a child span per item in a batch can multiply its span output by the batch size without changing its trace count, and a controller measuring traces per second will not notice. Measuring spans per second, or bytes per second at the exporter, closes that gap, at the cost of a measurement that the sampler cannot control directly.

The overall shape of the system is then a set of nested budgets: a storage budget at the bottom, a tail-stage retention budget above it, and a head-stage rate above that, each with a floor that guarantees visibility and a cap that protects the layer below. Adaptive control is the mechanism that keeps each layer inside its budget without an operator recomputing rates whenever traffic changes, and the reason it is worth the complexity is that the alternative is either a fixed rate that is wrong most of the time or a manual process that is wrong between adjustments. What the mechanism cannot do is make the sampled data self-describing; that remains an obligation on the instrumentation, which must record the probability it used so that anything computed later can be corrected.