Separating Injected Faults from Organic Failures in Telemetry

A fault injection that cannot be recognised in the telemetry produces two problems at once. During the exercise, the injected failure looks exactly like a genuine one, so whoever is watching has to reconcile two contradictory stories. Afterwards, the exercise’s failures remain in the historical record, where they distort error budgets, capacity decisions, and any model trained on past incidents. Marking the fault is not bookkeeping; it is what makes the exercise’s measurements usable.

What has to be attributable

Three things need separating, and they require different mechanisms. Requests need a marker, so that a failure caused by the exercise can be distinguished from a failure caused by anything else. Hosts or processes need a marker or a schedule, since some faults — a kill, a freeze, a memory limit — are applied to a target rather than to a request and carry nothing in the message path. Time needs a record, because the exercise has a start and an end, and everything between them is suspect.

Request-level marking works because the injector is on the request path. A request the injector generates, or one it has chosen to fail, can carry a field that propagates to every downstream hop, appears in the trace, and can be recorded as a label on the metric that counts it. Host-level marking has no such carrier; a killed process does not annotate anything, so the only workable record is an external one, stating which targets were affected and between which timestamps. The two mechanisms are complementary, and an exercise that uses only the request marker will fail to account for the consequences of a host fault on unrelated request streams, such as the retry traffic described in injecting failures at the RPC boundary.

Carrying a marker across every hop

The most reliable carrier is the distributed trace context, because propagation is already a requirement of tracing and every instrumented service forwards it. The W3C trace context defines a header whose value is a fixed sequence of fields: a version byte, a 16-byte trace identifier, an 8-byte parent span identifier, and a byte of flags. Any service that participates in the trace passes it on, so a marker encoded in the trace identifier’s associated attributes reaches every hop that records spans.

# the canonical example trace identifier from the specification, sent as a header
curl -H 'traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01' \
     -H 'X-Experiment: web-tier-partition-42' \
     https://api.example.internal/v1/orders

The second header in that example is application-defined. Ordinary header fields are extensible by design, and a custom field propagates as faithfully as any other hop-by-hop metadata as long as intermediaries do not strip or rewrite unknown fields. That is the catch: proxies, gateways, and cross-origin boundaries commonly drop or restrict unknown headers, so a marker that works inside the service mesh may disappear at the edge, and its absence is not evidence that the request was organic. Where propagation cannot be guaranteed, the marker has to be added at the outermost hop that can be controlled, or the exercise must rely on the ledger instead.

Sampling interacts with marking in a way that is easy to get wrong. Trace sampling decisions are usually taken at the entry point, so a marked request sampled out is invisible to the trace backend even though the marker was correct. Sampling based on the trace identifier is deterministic, which means a low sampling rate drops a stable fraction of the exercise’s traces; the analysis of an exercise with a small number of requests and a small sampling rate has nothing to correlate. Exercises should either raise sampling for marked traffic or use metrics, which are not sampled in the same way, as the primary record.

Query-time filtering instead of ingestion-time deletion

There is a strong argument for keeping the marked data and excluding it from the derived views. The raw stream is the only place where the exercise’s own cost can be measured afterwards: how much extra load the retries produced, how much of the error budget was spent, what the latency distribution looked like. Deleting at ingestion destroys exactly the evidence the exercise was meant to produce, and it does so irreversibly.

The cost of keeping it is tag cardinality. Every distinct marker value added as a metric label multiplies the number of time series, and a marker that embeds a per-run identifier makes the label unbounded. The workable compromise is a small, bounded set of marker values — an exercise identifier drawn from a fixed vocabulary, or a single boolean indicator for whether a request belonged to an exercise — combined with the trace data for higher-cardinality detail. Aggregates over the bounded label answer the capacity and budget questions; individual traces answer the diagnostic ones.

Exclusion at query time is also where a subtle fairness question lives. If the exercise’s requests are excluded from the service level indicators, the indicators describe a service that, for the duration of the exercise, was not the service anyone was using. That is usually the right call for budget accounting, and it should be paired with a second view that includes the marked traffic, so that the difference between the two views is itself reported. The difference is the cost of the exercise, expressed in the same units as the indicator.

Keeping the exercise out of the alert path

The purpose of alert handling during an exercise is to stop the exercise from generating a real response while leaving real failures able to do so. Blanket silencing of a service defeats the second half and is dangerous: an unrelated real incident that starts during the window is invisible, and the exercise’s own duration becomes a window of reduced observability.

Narrow suppression is the alternative. A silence or route that matches on the marker excludes only the exercise’s own alerts; anything unmarked still pages. That requires the marker to be present in the alerting data, which in turn requires the alert to be computed over a series that carries the label, and it means every alerting rule that is expected to stay quiet must be evaluated on a marked series. Rules derived from host-level faults can rarely be suppressed at all, because the fault produces ordinary traffic with no marker; for those, the honest approach is to notify the responders that an exercise is running and route the affected alert to the exercise’s channel rather than to the paging path, with an expiry set on the suppression.

Annotations on dashboards are the cheapest usable signal, and their limitation should be understood: they inform a human looking at the graph, and they change nothing about routing, evaluation, or page delivery. They are worth having anyway, because the first question asked when a graph looks strange during an exercise is whether the exercise explains it.

The injection ledger and why clock agreement matters

An exercise should write a record of what it did, when, and to what. The ledger does not need to be elaborate: a timestamped entry at fault start, at fault end, at each target change, and at the point the abort condition fired. It is the only source of truth for faults that never touched the request path, and it is what makes the timeline of a game day reconstructible after the fact.

The ledger is only as good as the clock that timestamps it. If the ledger’s clock differs from the telemetry backend’s clock, every correlation is offset by that difference, windows selected from the ledger will miss the beginning or the end of the fault, and a fault that moved the clock, as described in clock skew injection, can put them arbitrarily far apart. Writing the ledger with a monotonic timestamp for ordering and a synchronised wall-clock timestamp for correlation keeps both properties, and recording the two together is what allows a mismatch to be detected rather than silently trusted.

Markers also need a boundary. An internal marker field must not appear in a response body, a customer-visible artifact, or a public identifier, both because it leaks the existence of the practice and because a consumer may eventually depend on it. The marker belongs in the metadata plane, propagated between trusted hops and recorded in telemetry, and it should be stripped at the edge in the same place where internal identifiers are normalised.