Discriminating Tests and the Shape of a Suspect Set

An endpoint’s p99 latency doubles at 14:20 and stays doubled. The candidate causes are numerous: a deploy at 14:05, a schema migration, a shift in traffic mix, a noisy neighbour on the host, a garbage collection pause that grew with heap size, a downstream service timing out more often, a resolver returning stale addresses, a connection pool that stopped reusing connections. The usual response is to open dashboards one after another and look for something abnormal. That procedure has no stopping rule and no way to fail, which is exactly why it can absorb an afternoon without producing a conclusion.

A probe is a partition, not an observation

The useful abstraction is to treat the candidate causes as a set $H = {h_1, \dots, h_n}$ and to treat every diagnostic action as a function from $H$ to a set of possible outcomes. A probe that returns the same outcome under every hypothesis carries zero information regardless of how expensive it was to run. A probe that splits $H$ into two halves carries one bit. A probe that distinguishes all $n$ hypotheses in a single shot carries $\log_2 n$ bits.

This reframing changes what counts as progress. Reading a CPU graph is not progress in itself; it is progress only if “CPU is saturated” and “CPU is not saturated” predict different underlying causes. If both branches of that test lead back to the same short list, the graph was decoration.

The arithmetic is worth stating plainly. Sixteen equally likely suspects require four perfect binary probes in the worst case, since $2^4 = 16$. A probe that splits the set 15-to-1 still leaves fifteen suspects in the likely branch, so it is barely better than no probe at all. The best next probe is the one whose outcome is nearest a coin flip under current belief, because that is the split that maximises expected reduction. This is the same argument that makes binary search preferable to linear scan, applied to causes instead of sorted keys.

import math

def bits(n, k):
    """Information from a probe leaving k of n equally likely suspects on one branch."""
    p = k / n
    if p in (0.0, 1.0):
        return 0.0
    return -(p * math.log2(p) + (1 - p) * math.log2(1 - p))

print(round(bits(16, 8), 2))  # 1.00: a clean halving
print(round(bits(16, 1), 2))  # 0.34: a probe that names the favourite

There is a discipline that makes the partition explicit, and it costs a minute. Before running a probe, write down what each surviving hypothesis predicts the probe will show. If two hypotheses predict the same reading, the probe cannot separate them no matter how carefully it is executed. If a hypothesis predicts nothing at all, it is not a hypothesis yet; it is a suspicion, and it needs to be sharpened until it makes a claim that could be false.

Probes must be able to return no

The single most common defect in a debugging session is a probe whose result is already known. Asking whether the error rate is elevated, after an alert on the error rate has fired, tests nothing. Asking whether the host is healthy, when health is defined by the same checks that did not catch the problem, tests nothing.

A probe earns its cost only if at least one live hypothesis predicts a negative result. That is the falsifiability condition, and it is easy to violate accidentally because telemetry is built to answer questions about the ordinary case. A dashboard panel showing request rate is informative about capacity and nearly useless about a bug that affects one request shape.

The practical form of this rule is a prediction table. For a suspected connection-pool exhaustion, three hypotheses might be:

• the pool is fully checked out, so acquisition waits;
• the pool has free connections but the backend refuses new sessions;
• connections are acquired quickly and the latency is downstream.

These predict different values for pool wait time, for the count of open backend sessions, and for the ratio of time spent before the first byte of the response. Each of those is a discriminating probe. “Look at the latency graph” is not, because all three hypotheses predict high latency.

Correlated hypotheses and the axes that survive

Independence is the assumption that fails most often in real systems. Deploys, configuration changes, traffic shifts, and autoscaling events cluster around the same minutes because they are all triggered by the same operational rhythm. When two candidate causes always co-occur, no amount of telemetry collected on that axis can separate them; the partition does not exist in the data, and additional probes along the same axis only produce more correlated evidence.

The escape is to find an axis where the two causes diverge. If a build and a configuration change shipped together, the divergence might be a host that runs the new build with the old configuration, or a request class that exercises the changed code path but not the changed setting. If neither exists, the honest conclusion is that the current state of the system cannot answer the question, and the way forward is to construct the separating case rather than to keep measuring. The mechanics of constructing it are the subject of confounded changes in live systems.

A second kind of correlation is hierarchical. “The new build is slow” and “the new build is slow because the serializer regressed” are not two independent suspects; the second refines the first. A set containing both wastes a probe, because any test that separates the coarse hypothesis from its rivals also separates the refined one. Partition over mechanisms at one level of abstraction, then descend, which is the same shape as bisection over a commit range rather than over individual lines.

Pricing a probe in bits per minute

Once probes are understood as partitions, they can be compared on a ratio: information delivered per unit of cost. Cost has at least four components, and they are not commensurable, so the comparison stays qualitative but still discriminates.

Wall-clock cost is the obvious one. Reading a panel that already exists takes seconds. Adding a structured log line to a hot path requires a build, a deploy, and enough time at the new version for the affected requests to recur, which is minutes to hours.

Blast radius is the one most often ignored. A probe that redirects a fraction of traffic, disables a cache, or forces a garbage collection changes the system under study. When the probe is also the fix, the observation is contaminated: the symptom disappears and the cause remains unidentified, which is why “restart it and see” destroys evidence. The probe is still worth running sometimes, but it should be recognised as a hypothesis-eliminating action with a side effect, not as a measurement.

Resolution is the third. A boolean “is it slow” answers one question. A histogram of wait times answers a family of questions at the cost of more bytes and more attention. Higher resolution is not automatically better: a 20-bucket latency histogram can separate queueing from service time, while a single mean cannot, because the mean of a bimodal distribution describes no actual request.

Irreversibility is the fourth. A log line can be added and removed. A schema migration that rewrites a column cannot be un-run, so it belongs late in the sequence, after cheaper probes have narrowed the set.

A rough ordering follows: existing telemetry first, since its marginal cost is attention only; then read-only queries against production state; then reversible runtime changes such as sampling rate or a feature flag; then code changes deployed to a small cohort; then irreversible mutations. The ordering is not about importance. It is about preserving the option to run a better probe later.

When every probe agrees

The signature of an incomplete suspect set is uniform negative results: each probe returns what the leading hypothesis predicted, and the symptom persists. At that point more probes drawn from the same list are wasted, because the list is wrong. The failure lives somewhere the model of the system does not represent, and the correct move is to widen the boundary rather than to look harder.

Widening usually means moving down a layer: from the application to the runtime, from the runtime to the kernel, from the kernel to the network, from the network to name resolution or time. A service whose own timers show normal handling times but whose clients report long waits is describing a gap between two clocks, not a slow function. Recognising that the gap itself is the object of study is the shift, and clock domains and causal ordering covers the mechanics.

The same signature appears when the probe is measuring the wrong population. A tail latency problem that is invisible in a sampled trace set may simply be absent from the sample, since the sampler kept the fast requests. Evidence collected after the fact is conditioned on the collection rule, and that conditioning is itself a hypothesis worth checking before concluding that a cause has been ruled out.

What the method does not settle

Partitioning assumes the suspect set is finite and enumerable, which holds for designed systems and fails for emergent interactions. It also assumes that a probe’s outcome is stable, and some are not: a race that fires once per million requests will answer “no” to a probe run for one minute, and the negative result is real but weak. The strength of a negative observation scales with how much of the population it covered, which is why a probe over one percent of traffic is not the same evidence as a probe over all of it.

Finally, the cost model is asymmetric in a way that arithmetic hides. A cheap probe that returns an ambiguous answer consumes not just its own runtime but the interpretive work of deciding what the answer meant. Ambiguity is expensive, and a probe that produces a clean binary is often worth more than three that produce overlapping distributions.