How Sampling Rules Decide Which Evidence Survives

The investigation asks a question the collection system was not designed to answer. How often does this failure occur? What fraction of requests hit the slow path? Did the retry succeed? The data that would answer it was filtered, aggregated, or deleted before anyone knew the question existed, and the filtering is not random with respect to the answer.

Two samplers with opposite biases

Head sampling decides whether to record a trace when the request starts, typically by hashing the trace identifier and keeping a fixed fraction. Its defining property is that the decision is independent of the outcome, because the outcome does not exist yet. A one percent head sample of a million requests yields about ten thousand traces, and any rate computed from them is an unbiased estimate of the rate in the population, up to sampling noise.

The limitation is resolution. An error affecting one request in a hundred thousand appears, in expectation, zero times in a ten-thousand-trace sample. The sample is unbiased and uninformative at the same time, which is the distinction between accuracy and precision that matters most in incident work. The expected count is $\text{requests} \times \text{sample rate} \times \text{event rate}$; making the third factor small enough to matter makes the product small enough to vanish.

Tail sampling decides after the outcome is known, and keeps traces that are slow, erroneous, or otherwise interesting. This is the only practical way to retain rare failures at a useful count. It also destroys the ability to compute rates: the retained set contains the errors and a filtered subset of the successes, so the ratio within the set is a function of the sampling policy, not of the system. A tail-sampled collection of traces can support the claim “this error occurs, and here is what it looks like” and cannot support the claim “this error occurs at rate $r$”.

The distinction is easy to lose in a dashboard that presents both kinds of data side by side. A count of retained error traces is a count of samples, and it moves when the sampling rate changes, when the retention policy changes, and when traffic changes, in proportions that are not separable from the count itself.

Estimating what the sample can still support

When the selection probability is known per item, a weighted estimate restores the population quantity. The Horvitz-Thompson estimator divides each observation by its inclusion probability, so that an item kept with probability $1/100$ counts for a hundred items. Applied to traces, a head-sampled collection with a known rate supports unbiased totals by the same arithmetic: multiply each retained trace by the inverse of its sampling probability before summing.

Two conditions must hold. The inclusion probability must be known and must be greater than zero for every item in the population of interest. Tail sampling violates both when the policy is “keep any trace with an error”, because the probability of inclusion depends on the very attribute being estimated, and when the retention cap truncates by time, because the probability of surviving to the analysis depends on when the trace arrived.

The practical rule is to record the sampling decision with the record. A trace that carries the rate at which it was kept, and the policy version that kept it, can be reweighted later even if the policy has since changed. A trace that carries only its own content cannot, and the resulting ambiguity is permanent.

The third condition is the one that is almost always violated: the population must be the one that was asked about. Traces exist for requests that reached the instrumentation. Requests rejected at a load balancer, connections that failed before the application saw them, and requests handled by a process that died before its exporter flushed are all absent, and their absence is correlated with failure. A service that crashes under memory pressure exports no traces of its final seconds, and the trace set therefore underrepresents exactly the interval that matters.

Log levels as a sampler with a lag

Log filtering is the same problem with a slower feedback loop. A level threshold that excludes debug records reduces volume by orders of magnitude, and the excluded records are those most likely to describe the internal state of a failing operation. When the investigation needs them, the usual remedy is to raise the level, which changes the system’s behaviour (more I/O, more allocation, different timing) at the moment the behaviour is being studied.

Sampling of individual log lines has an additional hazard that trace sampling does not. A trace is a unit; a log line is not. Sampling lines independently breaks the correlation between the start and end of an operation, leaving half the story and no way to tell that it is half. Sampling by trace identifier preserves the unit, which is why the identifier is the natural key even when the records are logs rather than spans.

Thresholds also introduce a systematic skew in what counts as an error. A message logged at warning level by a library, and at error level by the application’s own wrapper, produces different volumes for the same event, and any comparison of error counts across services inherits the convention difference. This is not a sampling bias in the statistical sense, but it produces the same kind of mistaken inference: the numbers look comparable and are not.

Aggregation destroys the identity the investigation needs

Metrics are pre-aggregated by design. A counter accumulates, a histogram buckets, a gauge is sampled at scrape time. The aggregation is what makes metrics cheap to store and query, and it is also what makes them unable to answer questions about individual events.

The scrape interval is the first loss. A gauge sampled every fifteen seconds cannot show a spike that lasted three seconds, and an investigation that aligns a metric to a log timestamp is aligning two series with different effective resolutions, on clocks that are not the same clock. The bucket layout is the second: a histogram with a boundary at 100 milliseconds and the next at 250 milliseconds can say that a request took between those values and cannot say where. Quantile interpolation from buckets is an estimate with a known error band near the boundaries and no ability to recover the tail beyond the highest finite bucket, which is unbounded by construction.

Cardinality limits are the third and the least visible. A system that drops label combinations beyond a threshold removes exactly the high-cardinality values that identify a specific user, tenant, or endpoint, which are the values an investigation wants. The drop is usually silent, or reported in a self-metric that nobody monitors, so an analysis can conclude that a dimension has no effect when the dimension was never recorded.

Retention is the fourth. A fifteen-day retention window is long enough for routine work and too short for a failure discovered at day twenty by a customer report. Downsampling tiers compound it: the same logical series is available at full resolution for a day and at five-minute resolution for a year, and the coarse tier cannot distinguish a one-minute outage from a five-minute one. The relevant question is not how much data is kept but which questions remain answerable at each age.

Working with a census that is a sample

Three practices make post-hoc reasoning honest.

State the sampling frame before computing anything. The frame is the rule that determined which events could be present: the head rate, the tail policy, the level threshold, the retention window, the cardinality cap. Any rate quoted outside the frame is a guess wearing a number.

Prefer comparisons within a frame to comparisons across frames. Two services can be compared if both sampled at the same rate with the same policy; if one tail-samples errors and the other head-samples uniformly, the comparison measures the policies. Where a cross-frame comparison is unavoidable, the difference in frames should be quantified or the conclusion stated as ordinal rather than numeric.

Treat absence as a signal of its own. A gap in a series is ambiguous between “nothing happened”, “nothing was scraped”, and “the process that would have reported died”, and those three have different meanings for the investigation. Recording the reason for a gap — a missing scrape, a downed target, a dropped batch — converts an ambiguity into data, and it is the cheapest improvement available to a system whose telemetry is otherwise fixed.

The final constraint is that all of this applies to the evidence, not to the mechanism. A biased sample can still establish that a mechanism exists and can still supply the stack traces, error strings, and timing relationships that make it legible. What it cannot do is supply the weight of the evidence, and confusing the two is how a sample of one becomes a belief about the population. The distinction between establishing a mechanism and estimating its frequency runs through the reconstruction of an incident from what remains, where the two are routinely conflated.