Establishing Onset Before Correlating It With Changes

An alert fires at 14:20. The change that caused it may have shipped minutes earlier or days earlier, and the difference determines whether the investigation looks at the last deploy or at a schema migration nobody has thought about since Tuesday. Correlating the wrong window is expensive in a specific way: it produces a candidate that can never be falsified, because the investigator has no way to distinguish “this change caused it” from “this change happened near the alert”.

Detection lag and the shape of a ramp

Onset is the time the first affected request occurred. Detection is the time a human or a system noticed. The gap between them is composed of the time for the failure to affect enough requests to cross a threshold, the evaluation interval of the alerting rule, and any deliberate delay such as a “for” clause on the condition. A rate-based alert over a five-minute window on a failure affecting one request in a thousand will not fire until many minutes of traffic have passed, so a single spike in a low-volume service can be detected long after it ended.

The lag matters because it is not constant. A sudden total outage is detected within one evaluation interval. A latency regression of twenty percent may never cross an alert threshold and will be found by a user complaint hours later. Two failures with the same onset can therefore have detection times that differ by hours, and ordering investigations by detection time orders them by detectability rather than by cause.

The practical consequence is that onset must be estimated from data rather than taken from a notification. For a rate-shaped failure, the useful quantity is the failure fraction per unit time at a resolution fine enough that the ramp is visible: per minute for most services, per second for high-volume ones. For a latency-shaped failure, it is a percentile over the same buckets. Averages over five-minute windows hide onset by construction, because the first affected bucket is diluted by the healthy traffic inside it.

Finding the change point without knowing the answer

The estimation problem is a change-point problem: given a series of observations, find the time at which its distribution changed. The simplest method that works on operational data is a cumulative-sum style scan, which accumulates the deviation of each observation from a baseline computed before the suspected change and looks for a sustained trend in the accumulation. A single random fluctuation produces a local bump in the accumulation; a genuine change produces a slope that persists.

For debugging, the precision required is usually coarse, and that is liberating. Deciding whether onset is closer to 14:05 or to 09:00 does not require a statistically optimal estimator; it requires being able to see both candidates at once. Plotting the failure rate per minute over a twelve-hour window with a linear axis, without smoothing, is often sufficient, and the standard failure of this step is applying a smoothing function that was chosen for a dashboard rather than for the question.

Two mistakes recur. The first is estimating onset from a metric that the failure does not affect directly: request error rate can remain flat while a background job’s failure rate triples, if the background job’s errors are retried rather than surfaced. The second is using a series whose definition changed during the window, such as a latency histogram whose buckets were altered by a configuration change, which produces a step artefact that looks exactly like onset.

The series must also be aligned to a single time base before comparison. Timestamps in different formats and offsets, including the offset forms specified in RFC 3339, have to be normalised to one offset before they are compared; a comparison between a host reporting local time and a series reporting UTC produces a shift equal to the offset, which can move a finding across several plausible candidates.

Rollout windows and the meaning of a deploy time

Deployment timestamps are not points. A rollout that restarts a fleet sequentially takes minutes to tens of minutes, during which both the old and the new version serve traffic. The onset of a change-induced failure therefore follows the rollout’s progress through the fleet rather than the rollout’s start, and a failure that appears only when the last batch is updated is evidence about the fleet rather than about the code.

The clean way to see this is to split the failure rate by the version that each request was served by, which requires the application to report its version in its telemetry. Without that, the split has to be reconstructed from host restart times, which works when the rollout is sequential and fails when it is a rolling update across a pool with mixed versions at every moment.

Autoscaling complicates the same picture. A new instance joining at 14:05 with the new build changes the version mix, and the failure rate becomes a weighted average whose weights change over time. An onset that looks gradual may be a step in the mix rather than a step in the behaviour.

There is a second-order effect that turns correlation into coincidence. Configuration reloads, certificate rotations, feature flag flips, and dependency version bumps all happen on schedules of their own, so the changes near any given onset are numerous. A weekly deploy cadence with a daily configuration change means any hour has a substantial chance of containing a change of some kind, which is why a candidate found only by temporal proximity carries very little weight.

From correlation to a prediction that can fail

The step that converts proximity into evidence is deriving a prediction that the candidate mechanism makes and that a rival mechanism does not. If the cause is a timeout reduction, the affected population is requests that previously took longer than the new timeout, so the failure should be concentrated in the upper tail of the previous latency distribution and in the endpoints whose work is slowest. If the cause is a change to a single dependency client, the failure should be absent from endpoints that do not call that dependency.

That prediction is testable against data that already exists, which is the point. The test is a discriminating probe with a clean binary outcome: the affected population either has the predicted shape or it does not. A mismatch eliminates the candidate, and a match raises it above the other changes in the window without needing a controlled experiment.

When the prediction cannot be tested because the data does not distinguish the populations, the honest response is to record the candidate as unverified and move to a different probe, not to accept it because it is the only change in the window.

When there is no change to correlate

Reconstruction from configuration and cluster state is covered further in reading a diff for the mechanism it changed, and the case where the diff does not exist is the harder version of the same problem. The change may be a load increase rather than a code change: a new client, a marketing event, a scraper, or a retry storm from another system. Load-shaped causes are identified by onset correlating with a traffic property rather than a deploy, and the signature is that the failure rate is driven by the arrival rate of a specific request class rather than by time.

The change may also be outside all the systems under investigation: a downstream dependency changed its own behaviour, or a network path degraded. In that case onset is still the anchor that makes the search possible, because a dependency’s change log and a network path’s metrics can be queried in the same window, and the causal direction can be tested by the same population-shape reasoning.

What makes onset so productive is that it is a measured quantity, not an inference. Everything downstream of it, including which candidate wins, is a hypothesis, but the time itself is data, and it survives every subsequent revision of the theory.