Attribution, Turnover, and the Latent Measurement Error in Objectives

An objective is a measurement instrument, and like any instrument it has a calibration that can drift. The drift is not visible in the number the instrument produces, because the number remains a well-formed ratio over a well-defined query. What changes is the relationship between the ratio and the state of the system, and the change happens because the system it measures is not held still: endpoints are added, traffic mixes shift, instrumentation is rewritten, and the definition of what counts as an event quietly changes meaning.

This is the failure that objectives usually die of, and it is not the failure the literature concentrates on. A target that is too tight produces visible budget exhaustion and a negotiation. A target whose measurement has drifted produces a service that appears to be meeting a commitment while users experience something else, and the appearance is stable enough that nobody investigates.

The criterion is defined in terms of the interface, which moves

Consider an objective written as: requests returning a status code below 500 are successes, over the population of requests to the public API. The criterion references the status code as the record of failure. That was a defensible proxy when the API was a thin layer over a database. It stops being one when three things happen at once, none of which is recorded anywhere near the objective document.

A new endpoint streams results and returns 200 with a partial payload when a downstream dependency is degraded; the Content-Type is correct, the status is 200, the streaming connection terminates cleanly, and the client renders half a page. The objective counts it as a success. RFC 9110 is explicit that a 2xx status indicates the request was received and understood and the action was successfully completed, so returning 200 for a partial result is a statement that the action succeeded; if the action did not succeed, the status is wrong, and the objective is measuring the status rather than the outcome.

A second endpoint adopts a bulk operation that reports per-item failures inside a 200 response body. Whole-request success is now a coarse granularity: a request with one failed item out of a thousand contributes a success to the numerator and a user complaint to the support queue.

A third change moves validation to the edge, so malformed requests are rejected with 400 before reaching the service. Those requests leave the population entirely, and the edge’s rejection policy becomes part of the objective’s denominator without appearing anywhere in the objective’s text.

None of these is a bug in the measurement. Each is a change in what the measurement refers to. The indicator continues to compute correctly and its meaning has moved.

Schema and instrumentation turnover

The second source of drift is turnover in the code that produces the numbers. Instrumentation is not static: a metrics library is upgraded, a middleware is replaced, a new framework’s default request duration histogram replaces the hand-rolled one, the label set is tidied so that a cardinality problem goes away.

Each of these can change the value without changing the system, and the direction is not predictable. A middleware replacement might record duration at a different point in the request lifecycle — after routing rather than before, excluding or including TLS setup — which shifts the entire latency distribution. A histogram re-bucketing changes the interpolation error in the ratio, as covered in latency thresholds and tail budgets. A label rename splits one series into two for a period, and depending on whether the query matches on the old or the new label, the ratio may be computed over a fraction of traffic or may fail to match at all.

The last case is the most dangerous because it fails quietly. A query matching status=~"5.." continues to return a value; it just returns one that aggregates a different set of series. If the instrumentation change happened during a deploy that also introduced a defect, the change in the ratio mixes two causes, and the objective’s history is now a joined series of two different instruments. This is the observability equivalent of changing a ruler mid-measurement, and it is invisible unless the change is recorded alongside the objective.

The discipline that follows is that the objective’s definition — the exact query, the exclusion rules, the version of the instrumentation it assumes — is versioned as a configuration artifact, and every change to it is dated and attributed. Without that, a historical comparison is a comparison between two instruments.

Traffic mix is a weighting nobody chose

An objective’s value is a weighted average over the events in its population, with weights proportional to event counts. When the mix changes, the average changes even if every component is unchanged.

The clearest case is the slow endpoint. A service has two endpoints: a light read taking about ten milliseconds and a heavy report taking about two seconds. No request takes between the two. An availability objective over the whole API is unaffected by the mix. A latency objective with a 500-millisecond threshold is affected enormously: the light endpoint always meets it, the heavy one never does, so the ratio is exactly the fraction of traffic hitting the light endpoint. A shift in usage toward reports moves the objective from 99% to 50% with no change in the endpoint’s behaviour, and a shift away moves it back.

This is not a defect of the objective; it is what the objective says. The defect is in the interpretation, when the ratio is read as a property of the service rather than of the mix. Two corrections are available and they answer different questions. Restricting the population to endpoints with comparable latency characteristics makes the ratio a property of those endpoints, at the cost of needing one objective per class. Weighting the average by a fixed reference mix rather than by observed counts makes the ratio comparable over time but no longer describes what users experienced. Most teams want the first and implement the second by accident, when a dashboard’s per-endpoint breakdown is added on top of a global ratio and the two are read interchangeably.

Attribution is the substitute for a causal model

The reason drift is tolerable in practice is that objectives are not used to diagnose. They are used to answer whether the commitment was met, and when it was not, the investigation moves to traces, logs, and per-endpoint breakdowns. That division of labour only works if the breakdowns exist and are consistent with the aggregate.

Attribution fails in a specific way when events cannot be attributed to a single request. A queue-based system where a consumer processes messages from many producers, a batch pipeline where a failure invalidates an entire night’s output, and a shared cache where one tenant’s request poisons an entry used by others all produce failures that belong to a request other than the one that caused them. Measuring the objective on the producing request will show success; measuring it on the consuming request will show failure. The same event appears in two populations with opposite signs.

In such systems the indicator must be defined on the population where the harm appears, which is usually the consumer, and the producer’s objective must be defined on something else — message freshness, publish latency — rather than on the consumer’s success. The mistake that recurs is defining the producer’s objective on its own error responses and treating it as covering the consumer’s experience, which it does not, because the producer’s error responses are precisely the events that never reach the consumer.

Per-request trace data helps where it exists: a failed span can be joined to its parent and to the trace’s root, so failure attributed at the root is the user-facing failure and failure attributed at the leaf is the cause. The join requires the trace to be complete, which requires every service to participate in propagation and the sampling decision to be preserved across boundaries. When propagation breaks at one hop, the trace splits and the attribution becomes a join on time rather than on identity, which is a guess.

Auditing the instrument

A small number of checks catch most drift, and they are cheap because they compare the indicator against something that does not depend on it.

Count consistency. The number of events in the objective’s denominator should equal, within a stated tolerance, the count from an independent source: a load balancer’s request counter, a billing meter, a log-derived count. A persistent ratio between the two indicates an exclusion rule, a sampling bias, or a measurement point difference. A change in the ratio indicates drift.

Criterion sampling. Periodically sample requests that the objective counted as successes and inspect them: status, payload validity, latency at the client. A success that the client experienced as a failure is a criterion error, and it appears in samples long before it appears in the ratio.

Mix stability. Track the distribution of the population across the dimensions that drive the criterion — endpoint, tenant tier, protocol version, payload size class. A shift in the distribution is a warning that the aggregate is about to mean something different.

Instrument identity. Record the instrumentation version alongside the series, so that a discontinuity can be explained rather than attributed to the system.

These are analogous to the anomaly rather than the metric itself: each is meaningful only against its own history, and each requires that the history not be rewritten when the collection changes. That is why the exclusion and definition rules discussed in the SLI ratio: numerator, denominator, and measurement point should be applied and versioned at collection time, not recomputed at query time against archived raw data.

What remains unmeasurable

The residual is the part of the user experience that the objective cannot represent even with a perfect instrument: the request that succeeded after three retries inside the client, the page that loaded but showed data an hour old, the action that completed but was later reversed by a reconciliation. Each is a real outcome with no representation in a ratio over requests.

The response is not to enlarge the ratio until it covers everything, which is the composite-scoring problem treated in composite objectives and signal weighting. It is to accept that an objective covers a stated population under a stated criterion, that this is a proxy for user experience and not the thing itself, and that the proxy must be re-examined when the system under it changes. An objective that is never re-examined converges on the behaviour of the last time anyone asked whether it still measured what it said it measured.