In-Process Caches: Retained Heap and the Cost of a Hit

A cache in the application process is the fastest tier available: a hit costs a hash lookup and a pointer chase, with no serialization, no socket, and no context switch. That speed makes it tempting to treat as free, and the accounting that follows from that assumption is wrong by a large factor. An in-process cache retains the value, the key, the table slot that points at them, the eviction metadata, and any object graph the value happens to reference — and all of it lives in the heap that the garbage collector must trace on every collection.

The value is not the footprint

A cache entry written as “store this 200-byte object” typically retains several times that. The components are additive and each is easy to overlook.

The hash table holds a slot per entry, sized for a load factor near 0.75 in most implementations, so the table’s array is roughly 1.3 pointers per entry before any per-node object. A chained map allocates a node object per entry containing the hash, the key reference, the value reference, and a next pointer; on a 64-bit JVM with compressed object references these are 4 bytes each plus a 12 or 16-byte object header, rounded to 8-byte alignment — on the order of 32 bytes before the key and value are counted. A tree-based collision structure costs more per node but applies only to bins that overflow.

The key matters more than it looks. A string key retains its backing array plus header: a 16-character ASCII key is a header, a hash field, and a byte array of 16 bytes with its own 16-byte header, so roughly 60 bytes for a key that reads as 16. Composite keys built from several fields multiply this. Keys that are constructed per request and stored in the cache also pin whatever those objects reference if the key type is not a value type — a subtle leak in languages where keys can be arbitrary objects.

Boxing adds another layer in managed runtimes. A cached 32-bit integer stored as a boxed value costs 16 bytes rather than 4, and primitive-keyed caches that box on every lookup create garbage on the hit path, which turns the fastest operation in the system into an allocation. The remedy is a specialised map with primitive keys, or key types whose hashCode and equals avoid allocation.

The honest way to size a local cache is therefore to measure retained size for a representative entry — key, value, node, and amortised table slot — and multiply. On the JVM this is what a heap dump and a retained-size query produce; the total is several times the payload, depending on key size and object graph.

Bounded by count is not bounded by bytes

Most cache libraries accept a maximum entry count. That is a proxy, and a poor one when values vary in size: a thousand entries of 10 kilobytes and a thousand of 10 megabytes both satisfy “maximum 1000”. Several JVM caches offer a weight-based bound with a user-supplied weigher that returns a size per entry, and the eviction policy then works against total weight rather than count. Caffeine and Guava both expose this in the form of a maximum weight plus a weigher function.

The weigher must be cheap, because it runs on every insertion, and it must be accurate enough that the total does not drift. Returning value.length() for byte arrays ignores the node and key overhead; returning a constant per entry reproduces count-based bounding. A common compromise is to count the payload plus a fixed overhead per entry, which is within a constant factor of the truth.

Weight bounding does not help when the value is an object graph whose true size is not known at insertion time — a deserialized document with shared substructure, for example. Shared substructure also means that summing per-entry sizes overcounts, since two entries referencing the same interned object retain it once. Retained-size reasoning has to be done on the graph, not per entry.

Weak references, leaks, and the classloader

The classic leak in a long-lived local cache is not eviction failure but retention of something that should have died. Three patterns recur.

Keys that are strongly held by the cache while the cache itself is held by a long-lived object — a static registry, a singleton service — hold their values for the lifetime of the process. A cache keyed by a per-request or per-tenant object without bounding turns the cache into an accidental registry.

Listeners and callbacks registered by cached values keep the cache alive, or the cache keeps the listener alive. A removal listener that captures a reference to the object that owns the cache creates a cycle that a tracing collector handles but a reference-counting runtime does not; in either case the listener holds objects longer than intended.

Classloader retention appears where caches are per-deployment, as in application servers that redeploy a web application. A cache held by a static field in a class loaded by the application’s classloader retains that classloader, and through it every class and static it loaded, after undeployment. Weak references to keys help only when the key is the sole strong path to the classloader, which is usually not the case.

The purpose-built answer to the first pattern is a cache with weak or soft keys, or a bounded cache with an idle timeout, so that entries are dropped when nothing else references the key. Weak keys mean the entry survives only as long as some other part of the program holds the key object, which is exactly the semantics wanted for a cache that annotates objects already in memory. The danger is that the key can be collected between the lookup and the use of the value, which is why weak-keyed maps are a footgun in concurrent code.

Garbage collection is a latency participant

A local cache’s footprint changes the shape of collection, not just its cost. Large live sets get promoted to the old generation and are traced on every full collection; in a generational collector with a fixed heap, a cache occupying most of the heap means the collector has little room to work, and collection frequency rises. In collectors that compact, a cache of large objects moves rarely but occupies the regions that compaction has to preserve.

Allocation size matters independently. Some collectors treat allocations above a threshold as a separate category — the JVM’s humongous region threshold is half a region, and region sizes are typically 1 to 32 megabytes — so caching multi-megabyte values fragments the heap and triggers different collection behaviour than caching small ones. A cache whose entries regularly cross that threshold is a different operational animal from one holding a few hundred bytes per entry.

The practical consequence is that cache size should be chosen against heap headroom rather than against the machine’s memory. A process with a 4-gigabyte heap that dedicates 3 gigabytes to a local cache has left the collector with 1 gigabyte to operate in, and the tail latency of that process is now a function of the eviction policy’s effectiveness — the cache has become a GC problem as much as a lookup problem.

Duplication across instances

A local cache in a horizontally scaled service is N independent caches. Every instance holds its own copy of the hot set, so the total retained memory is N times the per-instance figure, and the effective hit ratio for the fleet is not better than for one instance — it is the same, assuming similar traffic. What changes is invalidation: an update must reach every instance, either by broadcast or by TTL.

Broadcast invalidation over a message channel is the usual compromise: each writer publishes the invalidated key, every instance removes it, an arrangement whose failure modes are those of invalidation by deletion at fan-out scale. A fire-and-forget publish/subscribe channel delivers at most once, so a dropped message leaves an instance serving a stale entry until its TTL. The TTL is therefore shorter than a remote-cache TTL would need to be, which increases the miss rate against the remote tier behind it. The layering is described in the discussion of tiers under hit ratio arithmetic and working-set sizing, but the memory arithmetic here is worth stating plainly: a local tier that absorbs 80 percent of reads still needs the remote tier to hold the full working set, so the local tier is an addition to the memory budget rather than a replacement for part of it.

Restart as a cold-start event

A local cache dies with its process. A rolling deploy that restarts every instance replaces a warm fleet with a cold one, and the remote tier or the source absorbs the difference. If the local cache was absorbing most reads, the restart is a full working-set reload; if the deploy is staggered and the local cache is the only tier, each restart sends a slice of traffic to the source. The mitigation is to warm before accepting traffic, from a recorded key list or from the remote tier, and to keep the per-instance working set small enough that reloading it is bounded work.