Cardinality Explosion and Where It Comes From

A metric name and a set of label pairs identify one series, and a series is the unit of storage, indexing, and query cost. The number of series is therefore the product of the number of distinct values each label takes, not the sum. This is the fact that makes metrics systems fall over, and it is stated so plainly in documentation that the failure still happens because the multiplication is invisible at the point where a label is added.

Adding user_id to a request counter does not double the series count. It multiplies it by the number of distinct users, which for a system with a hundred thousand active users is a hundred thousand times. The instrument still records one observation per request, the code still looks correct, and the cost appears days later as memory pressure in the collector and a query that times out.

The multiplication, stated exactly

For one metric with labels $l_1 \dots l_k$ taking $n_1 \dots n_k$ distinct values, the series count is $\prod_i n_i$ if every combination occurs. In practice combinations are sparse, since a request counter labelled by region and status will not see every region produce every status, so the real count is somewhere between $\max_i n_i$ and the product. The bound that matters for capacity planning is the product, because a workload change can fill in combinations that were previously absent without any code change.

Three label patterns produce most of the growth:

• Identity labels. User identifiers, session identifiers, request identifiers, trace identifiers, email addresses. These are unbounded by construction, since a new value appears with each new entity.
• High-dimensional attribute labels. URL paths including resource identifiers, error messages, SQL statements, command lines. Bounded in principle, unbounded in practice, because the set of possible strings is large and each new failure mode adds one.
• Unbounded numeric labels. Bucket sizes, retry counts, port numbers, HTTP status codes from a proxy that passes through upstream values. These grow slowly and then all at once, when a new upstream starts returning a status the label set has never seen.

The first category is a design error and is usually caught in review. The second is the common one, because a path like /api/v1/users/12345 looks like a useful label and is in fact one series per user. The third is the insidious one, because the growth is invisible until the day it is not.

What a series costs

The storage cost of a series is not the sample value. Each sample is a pair of a timestamp and a float, commonly encoded in a few bytes with delta or XOR compression, so the marginal cost of a sample is small. The dominant costs are the index entry, which is the full label set hashed and stored so that queries can find the series, and the in-memory head block that holds recent samples for every active series.

The index entry is roughly the size of the concatenated label names and values, plus the posting lists that map each label pair to the series that carry it. A series with five labels averaging ten bytes each costs tens of bytes of label data plus the postings, which is small per series and enormous at a million series. The head block is worse: recent samples for every series are held uncompressed in memory so that they can be appended to, and the memory is proportional to active series times samples per series, which is the scrape interval again.

The operational consequence is a ceiling that arrives suddenly. A system that comfortably handles a few hundred thousand series does not degrade gracefully at a million; it exhausts memory and restarts, and on restart it replays its write-ahead log, which takes longer because there are more series, and the replay may not finish before the next restart. This is the failure shape that makes cardinality a capacity-planning problem rather than a performance-tuning one.

Why a single unbounded label is worse than many bounded ones

Ten labels with ten values each are a hundred thousand combinations and are manageable. One label with a million values is a million series and is not. The difference is not the count, which is comparable, but the query behaviour.

A query that selects a specific label value uses the index to jump to the matching series, which is fast regardless of how many series exist. A query that aggregates across a label, such as sum by (region) over a counter labelled with a user identifier, must read every series and group them, which costs time proportional to the series count and produces a result that is mostly discarded. Worse, the aggregation has to materialise the grouping key for each series, so memory during the query scales with cardinality too.

There is a further cost in the index itself. The posting list for a label value that appears in a million series is a million-entry list, and intersecting several such lists is the expensive operation that query planners try to avoid. A label with few values has short posting lists that intersect cheaply, and the mechanics of that intersection are in Label Matchers, Posting Lists, and How a Series Is Selected. Cardinality concentrates the cost in exactly the place the index is supposed to save it.

Containment: what actually works

The measures that bound cardinality fall into three groups, and only the first is reliable.

Do not record the value. A label that identifies an entity belongs in a log line or a trace, not in a metric. The metric carries the count; the identifier goes where per-entity data is expected. This is a design decision made once, and it is the only measure with no ongoing cost.

Record the value in a bounded form. A URL path can be normalised to its route template, replacing identifiers with a placeholder, which turns an unbounded set into the number of distinct routes. The normalisation must be applied at the instrumentation boundary, because a query cannot recover the template from a concrete path. The same applies to error messages, which can be mapped to a small set of codes, and to command lines, which can be mapped to their executable name.

Bound the set at runtime. Some systems support a per-metric limit on the number of distinct label combinations, dropping or folding new combinations once the limit is reached. This converts an unbounded failure into a bounded one, at the price of silently losing data for the combinations that arrive after the limit. It is a backstop, not a design, and the loss is invisible unless the drop is itself counted, which requires a separate metric, which is itself subject to the same limit.

The measures that do not work are worth naming because they are commonly proposed. Sampling observations does not reduce series count, since every combination that occurs still creates a series. Aggressive retention shortens the time each series is stored but does not reduce the peak number of active series, and the peak is what kills the process. Moving to a system with a different storage engine changes the constant, not the asymptote: a columnar store with a per-series index still pays per series, and the multiplication is still a multiplication.

Detecting growth before it is fatal

The number of active series is itself a metric, and it is the one metric every deployment should record. Tracking it per job, per metric name, and per label name gives the derivative that matters: which instrument is adding series and at what rate.

The per-label breakdown is the useful one, because it identifies the label responsible. A label whose distinct value count is growing without bound is the culprit, and the count of distinct values is computable from the index without reading samples, which makes it cheap enough to evaluate frequently.

Alerting on the growth rate rather than the absolute count catches the problem during the period when the fix is still cheap. Alerting on the absolute count catches it when the process is already near its memory limit, which is usually after the deploy that added the label.

The interaction with histograms and with exemplars

Histograms multiply cardinality by the bucket count, so a service that adds a high-cardinality label to a histogram multiplies twice: once for the label and once for the buckets. The arithmetic is in Histogram Buckets, the +Inf Bucket, and How Quantiles Are Interpolated, and the practical implication is that latency histograms are the first place to look when series count jumps after a release.

A generated bucket layout trades a fixed boundary list for a resolution parameter, so the same latency distribution can be described without choosing boundaries in advance, at a memory cost proportional to the resolution; that trade is examined in Exponential Histograms and the Mergeable Bucket Layout.

Exemplars attach a sample trace identifier to a bucket, and they are deliberately excluded from the series identity so that they do not multiply it. A format that treated the exemplar as a label would reintroduce the problem it was designed to avoid, which is why exemplar storage is bounded separately and evicted on a different schedule from series data.

The broader point is that cardinality is not a property of the metric; it is a property of the label schema, and the schema is chosen by whoever writes the instrumentation. The containment that matters happens there, before the first sample is ever written.