Head-Based Sampling by Trace-ID Threshold

Sampling a trace at the point where it begins has one decisive property: the decision is made once, before any work is done, and it is carried to every participant. A service that receives a sampled trace exports its spans; a service that receives an unsampled one does no export work at all. The difficulty is making that single decision unbiased, consistent, and cheap enough to evaluate on every request.

The identifier as a uniform random variable

The trace identifier is 16 bytes, generated at the root of the trace. The W3C Trace Context specification requires it to be globally unique and generated without coordination between processes, and it forbids the all-zero value. Implementations draw it from a cryptographically secure generator, so its bits are indistinguishable from uniform.

That property is what makes threshold sampling work. Interpret the low 8 bytes of the identifier as a big-endian unsigned 64-bit integer, call it x. Then sample if x < p * 2^64 for a fraction p. Because x is uniform over the full 64-bit range, the probability of the comparison being true is p up to the granularity of the identifier, and the decision is independent of anything about the request, the service name, the endpoint, the latency, the tenant.

Independence is the point. Any decision correlated with the request would bias every aggregate computed from the sample. If sampling favoured slow requests, a latency percentile from the sample would be inflated. If it favoured one tenant, a per-tenant count would be wrong. A uniform hash of the identifier has no such correlation, so the sample is a uniform random subset, and multiplying a count by 1/p estimates the population.

The comparison is one instruction on a value already present in the request’s context. No randomness at request time, no per-request state, no coordination. Two services receiving the same trace identifier compute the same decision independently, which is the property that lets the decision be recomputed rather than carried, useful when a service is added to a system that never propagated the flag.

What the flag in the header actually says

Consistency across a distributed trace is normally achieved by propagation rather than recomputation, and the mechanism is a single bit in the traceparent header. The format is version-traceid-parentid-flags, with the flags field as two lowercase hex digits; the low bit is the sampled flag. A service that receives ...-01 exports spans and passes 01 downstream. A service that receives ...-00 does not export, and, critically, still passes the header downstream with the flag unchanged, because the downstream service may have its own reason to record something even when the trace as a whole is not being recorded.

The subtlety is that the flag is advisory in one direction and mandatory in the other. A participant may turn sampling on for a trace it received with the flag clear, this is what makes deferred or tail-based decisions possible. It may not turn it off for a trace it received with the flag set, because doing so would produce a trace with a hole in the middle: the parent exported, the child did not, and the reconstruction has a gap that is indistinguishable from a span that failed to arrive. A single bit supports only two states, and a fraction needs more. Systems that need to communicate the probability itself, so that a backend can compute correct population estimates from a sample, use an additional field in the vendor-specific tracestate header, or an explicit sampling probability attribute on the root span. The reason is arithmetic: if a pipeline samples at 10% in one region and 50% in another, a merged count is only correct if each span carries its own weight. Recording the probability alongside the span turns “count of spans” into “sum of 1/p”, and the sum is an unbiased estimate of the true count.

Where threshold sampling fails: the rare event

A uniform sample of 1% preserves the shape of the distribution of anything that occurs at least a few hundred times per sampling window. It destroys anything rarer. An error that occurs in one request out of a hundred thousand has an expected 0.01 occurrences in a 1% sample, which means most windows contain none, and the windows that contain one contain a single unrepresentative case.

This is not a defect of the implementation; it is the statistical consequence of sampling a heavy-tailed event population uniformly. The responses available are structural:

Sample errors at a different rate. A second, independent sampling decision applied to spans marked as errors, with a much higher probability, restores rare-event coverage. The sample is no longer uniform, so it can no longer be used for population estimates without weights, and the pipeline must keep the two strata separate or record per-span weights.

Keep the decision but defer the discard. The root makes the cheap decision, and the export path holds spans in a buffer long enough to see whether the trace contains an error, then makes a second decision. That is tail sampling, and it requires the trace’s spans to be co-located, the reason Tail Sampling and the Cost of the Decision Window is a topology problem as much as a statistical one.

Sample the head but always export a summary. The root records a minimal span for every request, an identifier, a duration, a status, and samples the detailed child spans. The summary is cheap because it has no children and few attributes, and it preserves the counts that the detailed sample would otherwise distort.

Each response moves cost and complexity somewhere else, and none is universally correct. The choice depends on which questions the trace data must answer: distributional questions tolerate a small uniform sample, and rare-event questions do not.

Deterministic versus consistent: two meanings of the same word

Threshold sampling on the trace identifier is deterministic: the same identifier always yields the same decision, regardless of when or where it is evaluated. This is what allows a decision to be recomputed after the fact and to match the original.

A different property, often conflated with it, is consistency across a set of related identifiers. Suppose a decision is made per-span on the span identifier rather than per-trace on the trace identifier. Each span is then sampled independently at rate p, and the expected number of spans retained for a trace of n spans is n * p. For a trace with a handful of spans, this yields fragments: three spans of a nine-span trace, with no parent for two of them. The trace is not partially useful; it is unusable for causal analysis, because the missing spans are exactly the ones that explain the retained ones.

Sampling on the trace identifier makes the decision once per trace, so the retained set is all-or-nothing. That is the entire reason the trace identifier exists as a distinct field from the span identifier, and it is why the field is 16 bytes rather than a per-process counter.

There is a cost to the all-or-nothing property: variance. A trace either contributes all n spans or none, so the estimator of total span volume has variance proportional to the trace size distribution rather than to the span count. For a workload where a few traces contain thousands of spans, the sample size in terms of spans fluctuates widely even though the number of traces is stable.

Choosing a rate and keeping it stable

Sampling rates are usually set per service or per endpoint, because the cost of a trace differs by orders of magnitude between a health check and a fan-out query. The rate must be stable over time, and this is the operational constraint most often violated: an operator who lowers the rate from 10% to 5% during an incident changes the population being measured mid-window, so any count compared across the change is comparing two different estimators.

A change in rate is safe only if the per-span probability is recorded, so that each record carries its own weight and a sum over a mixed window remains an estimate of the true total. Without weights, the honest statement is that counts before and after the change are not comparable, and the change should be annotated on any dashboard that spans it.

The rate also interacts with batching. A 1% sample of a service handling a thousand requests per second produces ten traces per second, each with several spans; at a ten-second flush interval a batch holds roughly a hundred traces, which is well below any transport limit. At a 0.01% rate the same service produces one trace every ten seconds, and the batch is nearly always flushed by the time trigger rather than the size trigger, the arithmetic in Batching and Flush Policy in Telemetry Agents applies unchanged, but the size trigger stops being the binding constraint.

What the root cannot know

Head-based sampling is a bet placed before the outcome exists. The root decides at request entry, and it has no access to the duration, the status code, or the presence of a downstream error. A trace that will turn out to be the most interesting one in the hour is sampled at the same rate as a health check that returns in a millisecond.

That asymmetry is the strongest argument for deferring the decision, and the strongest argument against deferring is that deferral requires every span of a trace to be held in one place until the decision can be made, which reintroduces the coordination that head sampling was designed to avoid. The tension between those two positions is the subject of the next step up in this cluster, and it is the reason many deployments run head sampling at a generous rate and tail sampling at a strict one, rather than choosing between them. The cost of the deferred decision, and the topology it forces, belongs to tail sampling; the property that survives from the head decision is that a rate applied at the root is still the cheapest way to keep the volume reaching the deferring tier within bounds.