Cardinality Control in Metric Pipelines

A metric name identifies a measurement; its label set identifies which instance of that measurement is being reported. Adding one label with a hundred possible values multiplies the series count by a hundred. Adding three such labels multiplies it by a million. The storage cost of a metric is therefore not proportional to its name count but to the number of distinct label combinations actually observed, and that number is chosen by whoever writes the instrumentation, often without knowing it.

The multiplicative structure

Series identity is the pair of a metric name and a complete label set. Every distinct combination is a separate time series with its own index entry, its own chunk of samples, and its own retention lifecycle. A histogram multiplies further: each bucket is a distinct series, so a histogram with fifteen buckets under a label set with a thousand combinations occupies fifteen thousand series before counting the sum and count series.

The arithmetic of label spaces is what makes the limit surprising. A service with a method label (ten values), a path label (fifty), a status label (ten), and a client label (a hundred) has a theoretical space of five million combinations. The observed count is far lower, because most combinations never occur, a given client calls a subset of paths. But the observed count is what determines cost, and it is determined by traffic diversity rather than by the schema. A long-tail workload that sees many distinct clients on many distinct paths approaches the theoretical bound, and the growth is gradual enough that nobody notices until the series count crosses a limit.

The decisive case is a label whose value is unbounded: a request identifier, a user identifier, a URL with a query string, an email address, an IP address with a port, a raw error message. Each distinct value creates a series that exists for as long as the retention window and is queried approximately never. One such label on a high-traffic metric can produce more series than every other metric in the system combined.

Where the control can be applied

There are four points at which a pipeline can influence the label set, and they differ in what they can see and what they cost.

At instrumentation. The producer decides which labels to attach. This is the only point where the semantic question “should this dimension exist at all” can be answered, and it is the cheapest place to fix the problem, because a label never attached costs nothing anywhere downstream. It is also the hardest to change, since it requires a code change and a rollout.

At the agent, by relabeling. Rules rewrite labels as samples pass: drop a label, rename it, replace its value with a constant, hash it, or drop the entire series when a condition matches. Relabeling is cheap, it is a hash table lookup and a few string operations per sample, and it is centrally configurable, which is why it is the standard control point for a pipeline that inherits instrumentation it cannot change.

At the collector, by aggregation. Sum or average samples across a chosen subset of labels, producing a new series with fewer dimensions. Aggregation reduces the count by exactly the product of the cardinalities of the dropped labels, and it is irreversible: the individual values are gone from the output.

At the backend, by rejection. A storage layer enforces a series limit per tenant or per metric and rejects new series beyond it. This is a backstop rather than a control, because rejection happens after the data has traversed the entire pipeline.

The order in a pipeline matters. Aggregating before relabeling drops labels that a relabel rule might have used to make a routing decision. Relabeling before aggregation is the correct order when the rule depends on a label that will not survive.

Dropping a label versus dropping the series

The two operations look similar and have opposite consequences.

Dropping a label from every series of a metric merges all series that differed only in that label. The merged series is the sum of the originals if the metric is additive, and the merge is exact: no information about the total is lost, only the breakdown. This is the safe operation, and it is the one to prefer.

Dropping a series, rejecting it entirely, removes its contribution from every aggregate computed downstream. If ten percent of series are dropped by a rule that matches a particular label value, then every sum over that metric is understated by the volume of those series, and no downstream consumer can detect the loss unless the pipeline counts the drops.

The distinction matters most for counters. Merging two counters that share a label set after the drop is correct only if both counters reset together, which they do not: two processes restarting at different times produce two reset events that the merged series sees as one. The merge is still usable for rate computation if the resets are rare relative to the window, and the error is bounded by the number of resets in the window. Merging is not usable for a strict total over a long period, where each reset loses the accumulated value of the series that reset.

Aggregation of gauges has an analogous problem with a different sign. Summing gauges that measure the same quantity across instances, a queue depth, for example, is meaningful. Summing gauges that measure a per-instance property, such as a memory limit, produces a number that corresponds to nothing. The metric’s semantics decide which aggregations are valid, and the pipeline has no way to know them unless they are declared.

Values that must be bounded rather than dropped

Some high-cardinality labels carry information that cannot be discarded: an HTTP path identifies the endpoint, and dropping it makes the metric useless for diagnosing which endpoint is slow. The solution is to bound the value space rather than remove the label.

Two techniques are common. The first is normalization: replace variable segments of a value with a placeholder, so that /users/12345/orders becomes /users/{id}/orders. The transformation is a regular expression or a template match, applied per sample, and its cost is the cost of matching against every value. The second is a top-k mapping: maintain a table of the values that have been seen, keep the k most frequent as themselves, and map everything else to a single overflow value. The table must be rebuilt periodically, and the boundary between the kept set and the overflow is not stable, so a value can move between them and appear as two different series over time.

Both techniques change the meaning of the metric. A normalized path is no longer the literal path, and a rate computed from it is the rate for the pattern rather than for one URL. The change must be documented at the point of use, because a dashboard that filters on a literal path silently returns nothing.

Enforcement and the failure it produces

A series limit enforced at the storage layer is a hard boundary, and the behaviour at the boundary is the design decision. Three behaviours are possible, and they differ in what an operator sees.

Reject new series, keep existing ones. The metric’s coverage freezes at whatever was present when the limit was hit. New instances, new endpoints, and new tenants produce no data at all, and the absence is indistinguishable from those entities not existing. This is the worst behaviour for diagnosis, because the pipeline looks healthy, the series that exist continue to report, while the entities that matter most during an incident are exactly the ones that are new.

Reject the entire metric. The metric disappears from the store. The failure is loud and immediately visible, and the blast radius is one metric rather than a subset of its series.

Sample or aggregate automatically at the limit. The pipeline drops dimensions it considers low-value, based on a heuristic such as cardinality contribution. The result is a metric that exists with degraded detail, and the degradation is silent unless the pipeline reports the substitution.

Whichever behaviour is chosen, the count of rejections must be exported as a metric of its own, on a series that is exempt from the limit. A pipeline that enforces a limit without reporting enforcement has replaced an observable problem with an invisible one.

Estimating cost before it is incurred

Series count is not the only cost; the cost per series depends on the sample interval and the retention window. At a fifteen-second interval, one series produces 5,760 samples per day. Five hundred thousand series produce just under three billion samples per day, which is a number that determines the storage tier and the query latency far more than the byte size of any individual sample.

An estimate before instrumentation is possible from the label schema alone, using the product of cardinalities as an upper bound and a measured distinct-value count as a lower bound. The lower bound is the useful one: a pipeline can count distinct label values for a candidate metric over a trial period and multiply by the number of labels that are not correlated with each other. The multiplication is the step that is usually skipped, and it is the step that turns a reasonable-looking schema into an unmanageable one.

The relationship between cardinality and the query side is the mirror image: a query that aggregates over a high-cardinality label must touch every series that matches, so the cost of a dashboard panel grows with cardinality even when the panel’s output is a single number. Shortening the collection interval multiplies the cost of every retained series in the same way, since each series contributes samples at a rate set by the interval rather than by the number of series.

Cardinality also interacts with the collection model. Under pull, the target decides its label set at scrape time and the collector sees whatever the target exposes; under push, the sender decides, and a label whose values are drawn from request data can multiply without any configuration change at all. The trade between the two models is set out in Push and Pull Collection Models and Who Owns the Sample Clock, and it determines which side of the pipeline a cardinality limit can be enforced from.