Composite Objectives: Compressing Multi-Signal Health into a Single Score

A service rarely has one objective. It has availability, latency, freshness, correctness, and throughput at minimum, each with its own ratio, window, and budget. When several of them are missed at once the status is unambiguous; the difficulty is the common case where availability is fine, latency is fine, and freshness has been degrading for six days. A single number that summarises all of them is attractive precisely because it would rank that against the alternative of a total outage.

The compression is possible and it is lossy in ways that are not obvious from the formula. Composite scoring is a design with a small number of parameters — which signals, how normalized, what weights, in what order aggregated — and each choice changes what patterns are visible. None of the choices is neutral, and the useful discipline is to know which patterns a given composite can and cannot distinguish.

Normalization destroys the units and the meaning

Before signals can be combined, they must be on a common scale. The standard choice is to express each as a fraction of the budget consumed: a signal at 100% of budget is normalized to 1, at 200% to 2, at 0% to 0. This is the same $B$ from error budget units and burn rate, and it has a defensible property: 1 means “spent exactly what was permitted” for every signal, regardless of target or window.

The alternative normalization, mapping each signal to a score between 0 and 1 by dividing the achieved ratio by the target ratio, is common and misleading. It says a service at 99.99% with a 99.9% target is at 1.0009 of target and a service at 95% with a 90% target is at 1.055, so the second appears healthier. The first has four times the headroom in permitted failures; the second is running far from its commitment in absolute terms. Whether that is right depends on whether the target is a commitment or a floor, and the two normalization choices answer different questions.

A third choice is percentile rank against peer services, which is meaningful for benchmarking and nearly meaningless for control, since health relative to peers does not say whether the objective is met.

The unit that is lost in every case is the count of affected users or requests. A budget fraction of 0.7 tells nothing about whether it was reached by 700 failures in a million requests or 70,000 in a hundred million, and a composite that hides the count cannot support a decision about urgency. Retention of the raw counts alongside the score is therefore not optional; it is what makes the score interpretable.

Weighting is a policy statement about acceptable failure

If the composite is a weighted sum of normalized signals, the weights determine the exchange rate between them. A weight of 0.5 on availability and 0.05 on freshness states that 1% of availability budget is worth 10% of freshness budget. That is a real claim about which degradation hurts more, and it is usually never made explicitly; weights get set to round numbers or divided evenly because the discussion is uncomfortable.

Two properties follow. First, weights are only meaningful relative to each other, so normalizing them to sum to 1 is a presentation choice. Second, the exchange rate should be derived from something external: the fraction of revenue attributable to each surface, the number of users whose workflow breaks, the contractual consequence of missing each commitment. Where no such derivation exists, an equal weighting is an admission of ignorance and should be labelled as such rather than presented as a measurement.

A composite with weights at one point in time also cannot express non-linearity. A total outage of availability is not exchangeable with any amount of freshness degradation, and any linear composite will say it is: enough freshness failure will dominate a full outage. The standard remedy is a hard gate: if any single objective is below a floor, the composite is capped regardless of the weighted total. The gate is a small amount of extra machinery that removes the worst class of absurd result.

Order of aggregation changes the answer

Signals can be combined across instances, across time, and across signals, and the three orders do not commute. The consequences are the same non-commutativity that affects ratio aggregation generally, applied one level up from a single indicator.

Averaging per-instance budget fractions and then averaging across signals weights a nearly idle instance equally with a saturated one. Averaging per-signal scores over time before weighting lets a brief, total failure in one signal be diluted by a long healthy period, which is exactly the case the composite was built to catch. Forming the composite per timestamp and then taking a mean over the window produces a different number again, and now a single bad minute contributes one data point among many rather than a sustained effect.

The defensible order for a control signal is: aggregate counts across instances first, form each ratio from summed numerator and denominator, normalize each ratio to budget consumption, then combine signals at each timestamp with the chosen weights, and only then take the worst value over the window or the time-weighted average — depending on whether the composite is used for alerting (worst) or reporting (average). Mixing those two purposes in one number is a common and avoidable source of disagreement.

Missing and stale signals are the hard part

Every composite has to decide what happens when one input is absent. The options are to drop it and renormalize, to treat it as zero consumption, to treat it as full consumption, or to hold the last value.

Renormalizing is the most common and the most dangerous. A signal goes missing because its instrumentation broke, which is likely to coincide with the incident that broke it, and the composite silently redistributes its weight to the healthy signals. A service whose correctness signal has stopped reporting will read as healthy on the remaining four. The failure mode is self-masking, and it is indistinguishable in the output from a genuine improvement.

Treating a missing signal as full consumption inverts the bias and makes every instrumentation gap a health emergency, which produces its own alert fatigue. Holding the last value is honest for a short gap and wrong for a long one. The compromise used in practice is an explicit staleness threshold per signal — often expressed in multiples of the signal’s own evaluation interval — after which the composite is marked indeterminate rather than reported. Indeterminate is a third state, and a system with only healthy and unhealthy states will collapse it into one of them.

The composite cannot replace its parts for alerting

A composite is useful for ranking, for status pages, and for deciding where to look. It is a poor trigger for paging, for a structural reason: an alert threshold on a composite is true when any weighted combination crosses it, and the mapping from a real failure to the condition is many-to-one. Diagnosing a page requires inverting the mapping, and that means the on-call engineer starts by decomposing a weighted sum.

The alerting rules in multiwindow burn-rate alerts are deliberately per-signal for this reason: each rule names the quantity, the window, and the threshold, and the alert carries the identity of what broke. A composite threshold named “service health below 0.6” carries none of that, and the first diagnostic step after it fires is to reconstruct the per-signal values that produced it.

There is a case for a composite alert at the top of an escalation chain — a single page when the service is broadly unhealthy in several dimensions at once, routed to whoever owns the composite — and it should sit above the per-signal pages rather than replace them.

Correctness does not compress

The signals that resist composite scoring are the ones that are not ratios of good to total: data loss, corruption, wrong results that were never detected as errors, security exposure. These have no natural denominator and no per-request success criterion, so they cannot be normalized to budget consumption without inventing a scale.

Folding them into a weighted sum is worse than leaving them out, because the sum implies an exchange rate: a small amount of corruption can be offset by excellent latency. The defensible arrangement is a composite over the measurable ratio-like signals, plus a separate, binary, non-exchangeable class of conditions that veto the composite entirely. That veto is the same hard gate described above, and its existence is the only thing that prevents the composite from being interpreted as a statement about overall safety.

The result of all this is that a composite objective is a summary of an accounting, not a replacement for it. It earns its place when its construction is documented, when its weights are justified, and when the underlying series remain the authoritative record. When any of those three is missing, the composite becomes a number that moves for reasons nobody can reconstruct, which is the precise definition of a metric that has stopped helping.