Hit Ratio Arithmetic and How to Size a Cache Tier

Hit ratio is the headline number of every cache, and on its own it is nearly useless. It says what fraction of requests were served from memory, not whether the cache is the right size, whether the source can survive its misses, or whether the misses that remain are the expensive ones. The arithmetic that makes it useful is simple, but it has to be done in the right direction: from the source’s capacity and the latency budget, backwards to a required miss rate, and then to a capacity that achieves it.

What the ratio measures and what it hides

The first problem is the denominator. A hit ratio computed over all lookups includes keys that were looked up once and never again, keys that were invalidated and immediately re-read, and keys belonging to a background job that runs nightly. Three different mixes produce three different ratios for the same cache and the same traffic to the source.

The second problem is the unit. Request-weighted and byte-weighted hit ratios differ enormously when a minority of large objects dominates the bytes. A cache that serves 95 percent of requests but only 60 percent of bytes is doing well for latency and poorly for bandwidth, and which one matters depends on whether the bottleneck is source requests per second or source bytes per second.

The third problem is the direction of the error. A cache that caches negative results inflates its hit ratio relative to the work it saves, since a negative hit replaces a lookup that answers only that the key is absent. A cache that serves stale data during a refresh inflates it further, since the refresh is a source read that is not counted as a miss. Neither is dishonest, but both mean the ratio is not the quantity to plan against. The quantity to plan against is source load: requests per second arriving at the backing store, and how that varies with the cache’s state.

The curve, not the point

Miss rate as a function of capacity is not linear, and it is not smooth in the way a capacity planner hopes. For traffic with a heavy-tailed key distribution — the Zipf-like distributions observed in web request streams, where the probability of the i-th most popular item falls roughly as 1/i^s — the miss rate falls steeply for the first increments of capacity and then flattens. Doubling a small cache removes a large share of misses; doubling a large one removes a small share.

Two features of the curve matter operationally.

The knee. Below the working set, each additional unit of capacity removes a predictable number of misses. Above it, the curve approaches the tail of one-shot requests that no cache can serve, and further capacity buys almost nothing. Sizing just past the knee is the usual target: enough to capture the recurring keys, not enough to hold the long tail.

The cliff at the other end. Shrinking a cache below the working set does not degrade gracefully. Eviction starts removing entries that would have been hit again, so the miss rate rises faster than the capacity falls, and each miss also displaces another entry. That is why a tier sized just below the working set performs much worse than the ratio of its capacity to the working set suggests, and why headroom is a performance parameter rather than spare space.

Measuring the curve requires varying capacity, which is expensive. A cheaper approximation is to log the key distribution and simulate: collect the access sequence for a representative window, replay it against simulated caches of different sizes with the policy in use, and read the curve off the simulation. Simulation is the only practical way to compare policies on the same trace, since the difference between two policies at the same capacity is workload-dependent and can be large on adversarial access patterns.

From ratio to source load

The conversion is one multiplication, and it is where the planning actually happens. If the application issues R reads per second at the cache, and the hit ratio is h, the source receives R(1 − h) reads per second. Each of those is also the read that populates the cache, so the count does not double: the source load is R(1 − h), and the capacity the source must provide falls from R to R(1 − h).

Inverting it: if the source can serve S reads per second at acceptable latency, the cache must achieve h ≥ 1 − S/R. A source that can serve a fifth of the application’s read rate requires an 80 percent hit ratio, and any workload change that pushes the required ratio above what the capacity delivers shows up as source saturation, not as a cache error.

The formula has a hidden term: the source’s capacity S is not constant. It falls when the source is under load, when a replica is down, and when a background job competes for the same resources. Planning with the steady-state S means a miss-rate increase of a few points can push the source past its knee, where its latency rises, which lengthens the miss path, which increases concurrent loads — the feedback loop described under coalescing and early expiration. A safety margin on S is the standard hedge, and it is a capacity decision as much as a caching one.

Latency arithmetic and the shape of the tail

Average latency is the wrong statistic for a cache. If a hit costs 1 millisecond and a miss costs 50, the average at a 90 percent hit ratio is about 6 milliseconds, which describes no request in the system. The tail does: one request in ten takes 50 milliseconds, so p90 sits at the miss latency and p99 is above it, since the slowest requests are often misses that arrived during a burst.

The composition matters more than the mean because of queueing. A miss occupies a source connection for 50 times as long as a hit occupies the cache, so at a fixed number of source connections the miss rate determines utilisation, and utilisation determines queueing delay. Doubling the miss rate roughly doubles source utilisation, which pushes the queueing term up non-linearly as utilisation approaches one. A cache that improves its hit ratio by five points near saturation can halve the source’s queueing delay, which is a far larger effect than the five points suggest.

The same arithmetic explains why a two-tier arrangement beats one large tier on latency without changing the overall hit ratio: the local tier serves its hits in microseconds, so the average latency is dominated by the fraction of requests that reach the remote tier, and the remote tier’s own hit ratio determines what reaches the source. The arithmetic is multiplicative: with a 60 percent local hit ratio and an 80 percent remote hit ratio, the source sees 0.4 × 0.2 = 8 percent of reads. That composition is the quantitative case for tiering, and it is why adding a small local tier in front of a large remote one moves the average latency more than enlarging the remote one.

What the curve does not hold still

A hit ratio measured last week is a statement about last week’s working set. Three processes move it.

TTL shortening mechanically lowers the ratio, because entries die before they are re-read. Raising TTLs raises the ratio and lengthens the staleness bound — the trade described in TTL expiry and the two ways an entry dies. A capacity plan that assumes a TTL policy is assuming a policy decision, not a workload property.

Key churn moves the curve. Versioned keys, per-request keys, and keys that embed a timestamp all reduce the reuse the cache can capture, because each write creates a new address and the old one becomes dead weight. A workload with heavy churn shows a hit ratio that never approaches the working-set bound no matter how much memory is added.

Traffic mix shifts the tail. A crawler, a backfill, or a new client version that reads different fields changes the distribution without changing the request rate, and the new keys are cold by definition. The ratio drops for a period whose length is the time to warm the new working set, and no capacity change shortens it.

Measurement choices that mislead

Four measurement decisions change the number more than any capacity change.

Counting hits by request rather than by source read makes a request that touches ten keys contribute ten outcomes, weighting the ratio toward whichever code path reads the most keys. Counting by source read makes the ratio directly convertible to load, which is why it is the better denominator for planning.

Excluding negative lookups from the denominator makes the ratio look higher on workloads with many misses for absent keys. Including them but excluding the entries they populate double-counts.

Reporting a fleet-wide ratio hides the per-instance spread that follows from hashing: with consistent hashing and enough virtual nodes the spread is small, but with a small number of nodes it can be large, and the hottest node is the one that fails first.

Ignoring time-of-day correlation between traffic and source capacity makes a peak-hour ratio look like a daily average.

The ratio is a derived quantity, and like any derived quantity it is only as good as the measurement it is computed from. The number worth putting on a dashboard next to it is the source’s utilisation, because that is the quantity the cache exists to control.