Symptom-Based Alerting and the Cause Signals It Replaces

A rule written against CPU utilisation fires whenever a machine is busy. A rule written against request latency fires whenever requests are slow. Both are alerts, and they differ in one respect that determines whether the page is useful: the first names a condition that may have no user-visible consequence, and the second names the consequence itself. The distinction is usually drawn as cause against symptom, and the practical argument for symptoms is not aesthetic. It is that the number of possible causes grows with the system, while the number of ways a service can fail its users does not.

The counting argument

A request-serving system has a bounded set of externally visible behaviours: it answers, it answers slowly, it answers incorrectly, or it does not answer. Every one of those can be measured at the boundary where the user meets the system, and the measurements are stable across redesigns. The internal conditions that produce them are unbounded. Every dependency, every queue, every cache tier, every garbage collector pause, and every configuration knob contributes a candidate cause, and the set changes with each deployment.

A rule per cause therefore scales with the wrong variable. A rule per symptom scales with the interface, which changes rarely. The asymmetry compounds in a second way: causes are shared. A saturated database produces slow responses in every service that reads from it, so cause-based rules fire once per dependent service while a symptom-based rule fires once per affected user-facing interface. The page count for a single incident is proportional to the fan-out of the dependency graph rather than to the severity of the incident.

There is a real cost on the other side, and it should be stated plainly. A symptom-based rule says that something is wrong without saying what. Diagnosis still requires looking at causes, so the cause signals have to exist and be inspectable; they simply do not have to wake anyone. The standard arrangement is that cause metrics are recorded, retained, and displayed on the dashboard that the responder opens after the page, while the paging decision is made from symptoms alone.

Distinguishing a symptom from a cause

The test is whether the signal can be defined without reference to the system’s internals. Request success ratio, request latency, and freshness of served data are all definable from the client’s position. Queue depth, thread pool occupancy, and replica lag are not: they are statements about how the implementation happens to be arranged, and a change of implementation can remove them without any change in behaviour.

A borderline case is a signal that is externally observable but only indirectly. A load balancer’s count of upstream connection failures is measured at a component the user never sees, yet it corresponds closely to “the service did not answer”. Such signals are usable when they are a faithful proxy for a user-visible outcome and unusable when they are a proxy for an internal mechanism that may be healthy. The same load balancer’s count of health check failures is a proxy for the second kind: a target can fail its check and still serve every request correctly, because the check exercises a path the traffic does not use.

A second test is whether the signal can be aggregated across instances without losing meaning. Symptoms usually can, because they are counted at a boundary where the counts add: total errors divided by total requests is the error ratio for the fleet. Causes usually cannot, because the interesting quantity is a per-instance condition, and a fleet-wide mean of a per-instance condition describes no real machine. A rule that averages CPU across a fleet will report a comfortable 40 percent while one machine sits at 100 percent, and the same averaging applied to latency will hide a slow subset behind a fast majority.

The third test is whether the signal has a natural threshold. A symptom carries its threshold in the objective: the acceptable error ratio is whatever the service level objective says it is, so the alert threshold is derived rather than invented, as described in Error Budgets and the Burn Rate That Consumes Them. A cause rarely has one. There is no principled answer to how much memory pressure is too much, which is why cause thresholds are usually set at a round number chosen after an incident and then left in place long after the condition it described has changed.

What a page must contain to be actionable

A symptom-based page names an effect the responder can verify and a quantity that is out of policy. Both parts are necessary. A page saying that the error budget is burning tells the responder the service is failing users but not where; a page saying that one endpoint is failing but not how badly relative to policy leaves the urgency undefined.

The useful shape is a statement of the form “the success ratio for interface X over window W is below target T, consuming budget at rate B”. Each element earns its place:

• The interface identifies what the users are doing and narrows the search to one request path.
• The window states the evidence period, which tells the responder whether this is a spike or a sustained condition.
• The target ties the number to the objective, so the threshold is not arbitrary.
• The burn rate conveys urgency in a unit that does not require knowing the objective by heart.

The page also needs a link to the dashboard carrying the cause signals for that interface, and it needs to be routed to someone with the authority to act on it. A page that arrives at a team which cannot change the failing component produces a handoff, and a handoff during an incident is a delay in the response with no compensating benefit. Routing decisions are their own problem; Page, Ticket, or Log Line treats the assignment of urgency as a separate design question from the detection of the condition.

The cost of removing cause alerts

Deleting every cause-based rule is not the goal, and doing so creates a specific blind spot. Symptoms are defined over interfaces that receive traffic. A component that is broken but not yet on any request path produces no symptom, and it becomes visible only when traffic reaches it, which may be during a failover when the remaining capacity is already reduced. Dependency health, certificate expiry, and replication status are conditions whose whole value is that they are detectable before they are symptomatic.

The workable split is by consequence. A cause signal that is a leading indicator of a symptom, with a known and short lead time, is worth alerting on because it converts a future outage into present work. A cause signal with no established relationship to any symptom is worth recording and not worth paging on. The distinction is empirical, and the evidence is a record of past pages: if a cause alert has fired repeatedly without a symptom following, the lead-time claim is unsupported.

The second cost is that symptom-based alerting depends on the symptom being instrumented correctly, and instrumentation errors are silent. A success ratio that counts only responses the server managed to send will exclude connection resets, timeouts, and requests dropped before reaching the handler, so the ratio stays high while users fail. The definition of the denominator is therefore part of the alert’s correctness, not a detail of the dashboard, and it deserves the same review as the threshold. Where the measurement point sits relative to the failure is what Fleet-Wide Alerting and the Per-Instance Rule has to settle before any aggregate ratio means anything.

Finally, symptom alerting does not remove the need to choose a window, and the choice is where most of the remaining difficulty lives. A short window catches a brief total failure; a long window catches a sustained partial one; a rule with one window cannot do both without either missing the first or paging on noise. The construction that resolves this uses two windows per rule and is treated in Multiwindow Burn-Rate Alerts and the Short Window as a Reset.