Observability · Cluster
Debugging Production
Hypothesis-driven investigation.
- 01 Discriminating Tests and the Shape of a Suspect Set How a probe partitions the set of candidate causes, why an uninformative probe is worse than no probe, and how to price a test in bits against the time it takes.
- 02 Bisecting a Deploy Range When the Fault Is Not Monotone Binary search over versions and inputs assumes the symptom persists once introduced; what that assumption costs, how delta debugging generalises it, and where the search stops being worth running.
- 03 Clock Domains and the Ordering of Events Across Hosts Why timestamps from two machines cannot be subtracted to get a latency, what NTP slewing and stepping do to a span duration, and how causal order survives without a shared clock.
- 04 Reproduction Fidelity and the Sources of Nondeterminism A failure that reproduces locally is not necessarily the failure that was observed, and the difference is a set of interleavings, clock readings, hash seeds, and externally timed events.
- 05 Reading a Diff for the Mechanism It Changed A bisected revision is a pointer to a change, not a cause. Separating default-value, ordering, and resource-lifecycle differences from cosmetic ones is what turns a pointer into a mechanism.
- 06 Establishing Onset Before Correlating It With Changes Detection time is not onset time. Finding the minute a failure actually began, and knowing the base rate of changes in that window, is what decides whether a correlation is evidence.
- 07 Confounded Comparisons Between Canary and Baseline A canary differs from its baseline in more than the code it shipped. Traffic mix, cache warmth, host hardware, and routing all shift at once, and a rare-event hypothesis needs far more requests than intuition suggests.
- 08 Telling Saturation Apart From Contention A busy resource and a blocked one look identical in a latency graph. Queueing arithmetic, run-queue depth, off-CPU time, and steal time are what separate them.
- 09 How Sampling Rules Decide Which Evidence Survives The telemetry left after an incident is a sample drawn by a rule chosen before the question was asked. Tail sampling, log thresholds, and retention turn the surviving data into a biased census.
- 10 Reconstructing an Incident From Partial Evidence A timeline assembled from logs, traces, metrics, and control-plane records is a model with assumptions. Separating what was observed from what the model implies is what keeps the reconstruction honest.