Low-Volume Objectives and the Distortion of Counting Qualifiers

A service handling four hundred requests a day, with an objective of 99.9% over twenty-eight days, has a budget of 11.2 failures for the entire window. Eleven failures. Any single incident that takes the service down for a minute can exceed the budget for the month, and a quiet month with ten random failures is indistinguishable from a perfect one at the resolution the arithmetic allows.

This is not a tuning problem. The ratio is a binomial proportion estimated from a small number of trials, and below a few thousand trials per window the estimate carries an uncertainty interval wider than the gap between any target and any plausible alternative. A service that reports 99.75% against a 99.9% objective may be perfectly healthy, and a service that reports 99.9% may be in trouble. The numbers are not measuring what they are being used to decide.

The estimator’s uncertainty is the first constraint

If $n$ requests each independently succeed with probability $p$, the observed success ratio $\hat{p}$ has standard error $\sqrt{p(1-p)/n}$, and a 95% interval is roughly $\hat{p} \pm 1.96$ standard errors. At $p = 0.999$:

• $n = 1{,}000$: standard error 0.001, so the interval is about $\pm 0.2%$, wider than the entire distance to a 99.5% objective.
• $n = 100{,}000$: standard error 0.0001, interval about $\pm 0.02%$.
• $n = 10{,}000{,}000$: interval about $\pm 0.002%$.

To distinguish 99.9% from 99.8% at conventional confidence, the window needs on the order of a million requests, since the difference is only a few standard errors wide at that scale. A service at four hundred requests per day over twenty-eight days has 11,200 samples, and its interval spans roughly a percentage point.

The practical mapping from volume to attainable resolution is: a target’s last significant digit should not be finer than the sampling noise. At 11,200 requests, a target of 99.9% invites reading changes that are inside the noise; 99%. At a million requests, 99.9% is meaningful; 99.99% is not. Choosing a target more precise than the data is the most common way to build an objective that no one trusts.

Options when the data is thin

Four responses are available and they are not exclusive.

Widen the population. If the objective is scoped to POST /v1/checkout and that endpoint receives four hundred calls a day, include related endpoints under the same contract or the same user journey. The trap is that widening changes what the objective claims: an objective over an entire API is met or missed by the aggregate, and a failing endpoint can hide behind volume from a healthy one. This is acceptable only if the aggregate is genuinely the thing being promised.

Lengthen the window. A 99.9% target over ninety days has 36,000 samples instead of 11,200 and a budget of thirty-six failures instead of eleven. The cost is latency: a regression detected today may not move the ninety-day ratio enough to act on for weeks. Lengthening the window stabilizes the accounting and blunts the control.

Change the target. Set the target to what the volume can resolve. A 99% target on 11,200 samples has a budget of 112 failures, which is a usable control variable, and the interval around the observed value spans about half a percentage point, which is the precision the data supports.

Change the unit. This is the interesting one and it is where counting qualifiers enter.

Counting at a coarser granularity

If the unit of success is a job rather than a request, the denominator becomes the number of jobs, and the arithmetic changes in both directions. A batch pipeline that runs every fifteen minutes has 2,688 runs in a twenty-eight-day window, which at 99.5% permits thirteen failed runs. That is more samples than the four-hundred-per-day request case and a target that can be met while tolerating visible failures.

The unit choice is not neutral, because a job that fails does not fail once. A pipeline with forty tasks per run, where a run fails if any task fails, has a per-run success probability equal to the per-task success probability raised to the fortieth power under independence. A task success rate of 99.9% yields a run success rate of about 96%. Setting the objective on runs and reporting the objective on tasks produces two numbers that differ by orders of magnitude in the failure budget, and both are correct statements about their own unit. Whichever unit the objective names, the other must not be used as a proxy for it.

The same phenomenon appears in time units. A service that must be serving correctly for each one-second interval of a day has 86,400 samples; a service sampled once per minute has 1,440. Time-based objectives at a one-minute resolution and at a one-second resolution are different measurements of different things, and the finer one is much more sensitive to short gaps that no user noticed.

Qualifier counting and the unintended denominator

A qualifier is a condition attached to the population: requests from paying tenants, requests for the primary API version, requests that were not served from a client-side cache. Each qualifier narrows the denominator and makes the sample thinner, and each is usually added for a defensible reason.

The aggregate effect is what causes trouble. A single service may be scored under a tenant tier qualifier, a region qualifier, a protocol-version qualifier, and a cache qualifier, and each narrows the population by a factor. A million requests a month across all tenants becomes a few thousand for the enterprise tier in one region on the current protocol excluding cache hits, at which point the target’s third digit is pure noise.

The pressure to add qualifiers also arrives from the reporting side. A dashboard request to exclude synthetic traffic is reasonable. A request to exclude traffic during a planned migration is less so, because the migration’s impact on users did not disappear. A request to exclude weekends, where usage is low and failure rates look high because a small denominator magnifies a small numerator, is a request to remove the volatility rather than the cause. The exclusion rules discussed in the SLI ratio: numerator, denominator, and measurement point apply here with a stricter test, since the denominator has less room to lose.

Summing slices reintroduces the variance

A common attempt to rescue a thin aggregate is to compute the objective per slice and then treat the slices as comparable. Ten slices at 99.5% each do not aggregate to 99.5% in any statistical sense, and the spread across slices is dominated by slice size. The smallest slice will show the most extreme ratio, because its sample is smallest, and it will therefore generate the most alerts. This is the low-volume problem reproduced at every level of the hierarchy.

The correct aggregation is to sum the numerator and sum the denominator and form the ratio once, as described for indicators in the SLI ratio: numerator, denominator, and measurement point. A per-slice view remains useful for locating a problem, but it is not a smaller version of the objective, and treating it as one means the slice with the fewest requests sets the operational tempo.

When to stop using a ratio at all

Some low-volume things should not be scored as ratios. A daily reconciliation job that must complete successfully, a certificate renewal that happens quarterly, a failover drill that runs twice a year: the notion of a percentage is a category error. These are covered better by absolute conditions — completed, not completed, within deadline, lateness measured in minutes — and by alerting on the condition directly rather than on a budget derived from a fraction.

The transition point is roughly where the number of permitted failures per window is very small. When the budget is under about ten events, the objective is a boolean in disguise, and the arithmetic of budgets is doing nothing except obscuring that. The alternative is not the absence of measurement; it is the presence of a different shape: freshness (age of the last successful run), completeness (fraction of expected records processed), and outcome of individual runs, each of which is meaningful at one sample per day.

These indicators also do not fit the burn-rate frame. A budget of eleven failures over twenty-eight days has no meaningful rate over an hour; the burn rate is either zero or over a hundred and the intermediate values carry no information. Alerting for such services is covered in multiwindow burn-rate alerts for the case where the ratio is usable, including why that mechanism degrades at low sample counts, and the alternative there is the same as here: alert on the absolute condition.

Documenting the choice

Whatever the volume, the objective’s documentation should state the sample size it operates on, so that any reader can judge how much a change in the reported number means. A target line without a sample count invites the reading that a movement of 0.05 percentage points is information. Providing the denominator makes the resolution visible, and it makes the trade-off between precision and volume explicit at the moment someone proposes tightening the target by an order of magnitude without adding any traffic.