Detecting Slow Growth in a Long-Running Process

A leak that adds a few kilobytes per request produces no failure for a long time. It does not trip an allocation-failure alarm, because no single allocation fails; it does not show in a heap profile as a large row, because each row’s total is small; and it does not correlate with any error rate, because nothing errors until the process dies. The only signal is the shape of the footprint over a long interval, and reading that shape correctly requires separating three things that produce similar graphs.

Three graphs that look the same

A resident-size graph over days shows a slow rise in all three of the following situations, and they require different responses.

A genuine leak. Live objects accumulate because references are retained beyond use. The rise continues after the workload stops; the live set at idle never returns to its starting value.

Allocator sawtooth with a rising floor. The program allocates and frees in a cycle (per request, per batch, per garbage collection), and the allocator retains the peak of each cycle. The graph is a saw tooth whose teeth are bounded and whose floor is stable. If the floor rises, either the peak is rising because the workload’s working set is rising, or the allocator is retaining more each cycle.

Legitimate growth. A cache warming, an index loading, a connection pool filling. The rise is monotone, bounded by a known ceiling, and stops.

Distinguishing them requires more than the resident series. Two additional signals do most of the work: what happens when the workload stops, and what the live-object accounting says independently of the resident size.

The idle test

The cheapest discriminator is to stop the workload and watch. A process that has finished its input and is doing nothing should return to a stable footprint. If the footprint falls to a floor and stays, the growth during the workload was working-set or allocator retention, both of which are bounded by the peak. If the footprint does not fall, or falls partway and then resumes rising with no work being done, the retained memory is live in the program’s own accounting.

The test has two failure modes that make it easy to misread. First, a program with a background timer that allocates, such as a metrics flush, a periodic reconnect, or a cache expiry sweep, is never truly idle, and its footprint rises slowly with no user traffic. Second, the resident size does not fall when memory is freed: RSS stays at the high-water mark until the allocator releases pages or the kernel reclaims them. An idle test that relies on RSS therefore shows no fall even when the live set has collapsed, which produces a false leak signal.

The fix for the second failure is to measure the live set rather than the resident size. If the program has an allocation tracing layer, the traced in-use bytes are the live set and they do fall on idle. If it does not, the closest available proxy is the anonymous charge from the group counters, which excludes page cache but still includes allocator retention. Neither is exact, and the difference between the live set and the resident size is itself the quantity described under allocator metadata and overhead .

Estimating a slope that means something

Once a series exists, the question is whether its slope differs from zero by more than the noise. The series is noisy for reasons that are structural: the sawtooth from per-request allocation, the step changes from page release, the plateaus from caches reaching their ceiling. A comparison of the first and last samples is dominated by whichever phase they happened to land in.

Two approaches are worth using together.

Regression over a windowed series. Fit a line to the last several hours of samples, taken at a regular interval. The slope is in bytes per unit time, which converts to a projection: at the observed rate, how long until the process reaches its limit. The projection is the actionable number, and it is what distinguishes “this grows” from “this will kill us on Thursday”. The fit needs enough samples to average out the sawtooth (a few dozen at minimum), and it needs the samples to be taken at the same point in the workload cycle, or the phase of the cycle becomes a periodic term that the line absorbs as noise.

Comparison against the post-restart baseline. Every restart resets the process to a known state. The footprint after a restart, measured at the same point in the startup sequence and after the same warm-up, is a baseline. Growth relative to that baseline, over successive restarts, is the leak’s contribution accumulated over each process’s lifetime. A series of restarts that each reach a higher post-warm-up footprint at the same uptime is strong evidence, because the confounders (workload shape, cache warmth, allocator state) are held roughly constant.

The second method is more reliable and less convenient, since it requires restarts. It is also the only method available to a process with no instrumentation at all: comparing the footprint at a fixed uptime across many runs needs nothing but the resident-size counter.

# Slope of a footprint series by least squares, with a projection to the limit.
import statistics

def slope_per_hour(samples):
    # samples: [(hours_since_start, bytes), ...] in time order
    xs = [s[0] for s in samples]
    ys = [s[1] for s in samples]
    mx, my = statistics.fmean(xs), statistics.fmean(ys)
    num = sum((x - mx) * (y - my) for x, y in samples)
    den = sum((x - mx) ** 2 for x in xs)
    return num / den if den else 0.0

def hours_to_limit(samples, limit_bytes):
    b = slope_per_hour(samples)
    if b <= 0:
        return None                      # not growing faster than noise
    return (limit_bytes - samples[-1][1]) / b

Least squares over a series with a sawtooth returns a slope that is a compromise between the teeth’s upward and downward excursions, which is what is wanted. It is not tolerant of a single outlier, since one sample taken during a spike pulls the line, so the series should be trimmed of obvious spikes, or the fit done on medians of buckets rather than on raw samples.

Live-set accounting by epoch

The strongest signal available without a full tracing collector is a per-epoch live-byte counter, sampled at a fixed cadence. The construction: the allocator, or a wrapper around it, maintains a counter of bytes currently allocated. At each sampling tick, record the counter alongside the tick’s work measure: requests served, records processed, transactions committed.

The ratio of live bytes to work done is the quantity that matters. A program whose live bytes grow proportionally to work done is retaining per unit of work, which is the signature of a leak. A program whose live bytes grow while work is flat is retaining per unit of time, which points at a background path. A program whose live bytes are flat while work grows is simply doing more, and its footprint is bounded.

The ratio also localises the leak in time. A leak that appears only after a particular kind of request produces a step in the ratio at that request; a leak that is present from the start produces a constant slope. Neither is visible from the resident-size series alone, which is why the counter and the work measure have to be sampled together rather than separately.

The counter itself has to be cheap. An atomic increment on every allocation is a shared cache line and a serialising operation on the hot path; the usual implementation keeps per-thread counters and sums them at the tick, accepting that the sum is not a consistent snapshot, being the sum of per-thread values read at slightly different instants. For a trend measurement over hours, the inconsistency is immaterial; for a claim about a single instant, it is not.

Where the growth is, once it is established

Establishing that growth is real narrows the search but does not identify it. The attribution steps follow from what the growth’s shape says.

Growth that survives idle and is proportional to work done points at a per-request retention. The allocation stacks that grew between two snapshots, taken hours apart and diffed by call path, identify the site. This is the growth-diff over allocation stacks, and its weakness is that a slow leak’s growth per snapshot is small relative to the noise from caches and pools. Diffing over a long interval, rather than a short one, is what makes the leak’s contribution exceed the noise.

Growth that survives idle but is flat in work points at a background path: a timer, a reconnect, a queue that is drained by something slower than it fills. The mapping breakdown separates this from a per-request leak only if the growth is in a distinct region: a queue with its own mapping is visible as a single growing anonymous area, whereas a queue allocated from the general heap is not.

Growth that does not survive idle is not a leak. It is the working set or the allocator’s retained peak, and the response is either to accept it or to change the allocator’s release policy so that the footprint tracks the live set more closely. The second is a real trade: releasing pages aggressively costs page faults on the next allocation, and the choice between a higher steady footprint and a higher allocation cost is a decision about which resource is scarcer.

The limits of what a footprint series can prove

A footprint series can establish that a process retains memory over time. It cannot establish why, and it cannot establish intent. A cache that is supposed to grow and a leak that must not grow produce the same series until the cache reaches its ceiling, and the only difference is whether the ceiling exists.

That is why the useful monitoring design pairs the series with an expectation. The expectation is a ceiling per process, derived from the workload’s known working set, and the alarm is on the projection rather than on the current value: a process projected to reach its limit before its next scheduled restart is a problem now, while a process that has grown to half its limit and stopped is not. The projection requires a slope, the slope requires a series, and the series requires sampling at a cadence that is regular enough to fit. That is the entire measurement discipline, and the reason a single-threshold alarm on resident size is both too noisy and too late.