Fleet-Wide Alerting and the Per-Instance Rule

Every alerting rule has an aggregation level, whether or not it was chosen. A rule evaluated over per-instance series has one alert per instance. A rule that sums across the fleet has one alert for the fleet. The level determines two things at once: what the smallest detectable failure is, and how many notifications a single incident generates. It is the most consequential setting in a rule and the one most often left implicit by writing the expression without thinking about its result shape.

What the evaluation level can see

An aggregate can only represent what survives the aggregation. A fleet-wide error ratio is the sum of errors divided by the sum of requests, so a single instance failing completely is diluted by however much healthy traffic the others contribute. With ten instances of equal traffic, one instance at a hundred percent errors produces a fleet ratio of ten percent, which is a large number and easy to detect. With a hundred instances, the same failure produces one percent, which is indistinguishable from ordinary variation in many services. The aggregate hides exactly the failures that redundancy is supposed to absorb, which is often the right thing and sometimes is not.

Per-instance evaluation has the opposite failure. It sees every instance-level condition, including ones with no consequence, because the service is redundant and the condition is being handled by the remaining capacity. A rule that fires on any instance exceeding a threshold will fire during every rolling deployment, every cache warmup, and every garbage collection pause, and the page count reflects the fleet size rather than the number of incidents.

The right level follows from where the loss of redundancy becomes a user-visible problem. For a service with three replicas behind a load balancer, one unhealthy replica may be invisible and two unhealthy replicas are half the capacity, so an aggregate and a count-of-unhealthy condition are both useful while a per-replica condition is not. For a sharded store where each shard serves a disjoint set of keys, a single unhealthy shard is a user-visible failure for the keys it owns, so per-shard evaluation is not an option; it is the only level at which the symptom exists.

Counting rather than averaging

For systems with a redundancy factor, the useful aggregate is usually a count of degraded members rather than a mean of their condition. The count is directly interpretable as remaining capacity, and it is stable against the fleet growing or shrinking, which a mean is not.

PromQL makes the shape explicit. The expression below returns one series per shard, and the comparison retains only shards above the threshold:

# Shards whose error ratio exceeds 5% over the last 10 minutes
sum by (shard) (rate(requests_total{code=~"5.."}[10m]))
  / sum by (shard) (rate(requests_total[10m])) > 0.05

Wrapping that in count(...) > 2 converts it into a rule that fires when more than two shards are simultaneously degraded, which is a statement about remaining capacity. Wrapping it in max(...) > 0.25 instead fires when any shard is severely degraded, which is a statement about the worst user experience. Both are legitimate and they answer different questions. The first is a fleet-capacity alert and the second is a worst-case alert, and a service usually needs one of them rather than both.

Two details decide whether these expressions mean what they appear to. The first is that sum by must be applied to both numerator and denominator before dividing, otherwise the division matches series pairwise and produces one ratio per series, which is a different quantity. The second is that the resulting alert’s label set determines its identity in the state machine. Keeping shard in the output means one alert instance per shard and therefore one notification per shard unless the router groups them; dropping it via an outer aggregation means one instance for the fleet, at the cost of no longer knowing which shard caused it. An alert that must name the affected subset should keep the label and rely on grouping to control notification volume; an alert that only needs to say the fleet is in trouble should aggregate it away.

Cardinality, both of the metric and of the alert

The number of series a rule evaluates is the product of the label dimensions its selector matches. A rule over a per-instance metric on a fleet of thousands of instances, with a second label dimension for endpoint, evaluates a series count in the millions at every evaluation, and that cost is paid whether or not any of them is unhealthy. The alert output has its own cardinality: an alert instance exists for the duration of the condition, and each pending and firing instance is state the system must retain and the router must handle.

Both cardinalities are bounded in practice by deciding which dimensions are subject to alerting at all. Per-endpoint error ratios over a fleet are usually not worth alerting on individually, because the endpoints share fate: when one degrades, the others usually do too. Per-endpoint detail belongs on the dashboard, evaluated on demand, while the rule pages on the aggregate. The general principle is that alerting watches the small number of dimensions along which failures are independent, and dashboards carry the rest.

A second bound is the use of recording rules. A recording rule that precomputes a per-shard ratio on a fixed schedule turns an expensive per-evaluation computation into a cheap lookup, and it also fixes the aggregation level in one place rather than repeating it in every rule. The cost is indirection: the alert rule now reads a derived series, and its definition of the underlying quantity lives elsewhere, which is a real obstacle when someone later has to reconcile a page with the raw data. It also makes the alert depend on a series that a failed recording evaluation will not produce, which is the absence case covered in Alerting on Absent and Stale Data.

Redundancy changes what a symptom is

Aggregation level also interacts with the objective. A service with a 99.9 percent objective measured at the request boundary is already absorbing instance-level failures in the indicator itself, because the failed requests that a healthy fleet retried successfully do not appear as errors. Alerting per instance on top of that double-counts: the alert fires on the mechanism while the indicator, which is the thing that defines the budget, is unaffected. The consistent arrangement is to alert at the level where the indicator is measured, which is the same boundary argument made in Symptom-Based Alerting and the Cause Signals It Replaces, and to keep instance-level conditions visible but not paging.

The exception is redundancy that is not real. Three replicas behind a load balancer that routes to all three are redundant only while the load balancer can detect a failure and route around it; if its health check is slow, requests are lost during the interval in which the unhealthy replica is still in rotation. Requests lost in that window do appear in the indicator, which means the aggregate sees them, so the aggregate remains the correct alerting level. What a per-instance rule would add is earlier detection of the underlying condition, at the cost of a page per instance per event. That trade is a deliberate one and worth making only where the lost-request volume during the detection interval is large.

Choosing and revisiting the level

A workable procedure is to start from the user-visible failure and ask what must go wrong for it to occur. If the answer is “any single instance”, the rule belongs at the instance level and the service has no effective redundancy, which is worth knowing. If the answer is “more instances than the capacity margin allows”, the rule belongs at a count level. If the answer is “only a fleet-wide condition”, the rule belongs at the aggregate level, and the per-instance signals belong on a dashboard.

The chosen level should be revisited when either of the two things that determine it changes: the redundancy factor of the service, or its traffic distribution. A fleet that has grown from ten instances to two hundred has changed the dilution factor of its aggregate by an order of magnitude, and thresholds chosen for the smaller fleet will now be either silent or hyperactive depending on which direction they were chosen in. This drift is invisible in the rule file, and the only signal that it has occurred is that the rule no longer fires when it should or fires when the service is healthy.

The same reasoning applies in the other direction when the fleet mixes sizes. A mean of per-instance ratios over a heterogeneous fleet is dominated by the instances with the least traffic, because their ratios are computed from the smallest samples and therefore vary the most. A ratio of sums has no such distortion, which is another reason to prefer it. Where an aggregate must be formed from per-instance ratios, weighting by request volume reproduces the ratio-of-sums result and is worth doing only when the raw counts are unavailable.