Scrape Interval, Lookback Delta, and the Resolution of a Rate

Metrics are sampled, not streamed. A collector asks an endpoint for its current values at a fixed period, and everything the query layer does afterwards operates on that discrete sequence. The sampling period therefore sets a hard ceiling on the time resolution of every derived quantity, and the ceiling is not a smooth degradation: below a certain frequency the signal does not merely blur, it changes shape and can appear to oscillate at a frequency that was never present.

The sampling period is a low-pass filter with a sharp edge

A series sampled every $T$ seconds can represent variations with period no shorter than $2T$. Frequencies above $1/(2T)$ are not attenuated gradually; they are folded back into the representable band and appear as a lower frequency. For a request rate that peaks every 30 seconds and a scrape interval of 15 seconds, the peak is captured. At a scrape interval of 20 seconds the same peak is sampled at a slowly drifting phase and the resulting series shows a slow amplitude modulation whose period is the beat between the scrape period and the signal period, not the signal’s own period.

The folding rule is arithmetic. A signal component at frequency $f$ sampled at rate $f_s$ appears at the aliased frequency $|f - k f_s|$ for the integer $k$ that brings the result into $[0, f_s/2]$. A 40-second cycle sampled every 15 seconds becomes a component at $|1/40 - 2/15| \approx 0.1083$ Hz, a period of about 9.2 seconds, which looks like a fast oscillation rather than a slow one. The visible artefact in a dashboard is a wave that moves when the scrape interval is changed, which is the diagnostic signature of aliasing rather than real behaviour.

Real metrics are not band-limited. Traffic varies on scales from sub-second bursts to daily cycles, so aliasing is always present to some degree. What the sampling period controls is where the fold happens, and therefore which real variations are faithfully represented and which are converted into spurious ones.

Two constraints on the query window

A rate query over a range shorter than the sampling period has at most one sample and cannot produce a value. Between one and two periods it has two samples and produces a value, but the result is dominated by the extrapolation described in Counter Resets and the Extrapolation Inside rate(), because a single interval’s worth of jitter is being stretched to cover the whole window.

The practical minimum is four samples, which is three intervals of coverage. This is not an arbitrary convention; it comes from the reset detector needing at least two comparisons to distinguish a genuine decrease from noise, and from the edge extrapolation needing enough interior samples that the extrapolated fraction is a minority of the total span. With four samples the extrapolated portion is at most about a quarter of the window.

There is a second constraint in the opposite direction. A long window smooths away real variation, and the amount of smoothing is exactly the window length: a rate computed over five minutes cannot show a spike that lasted thirty seconds with more than a fraction of its true amplitude. Choosing a window is therefore choosing which frequencies to display, and a dashboard that shows the same signal at one-minute and one-hour windows is showing two different signals that happen to share a source.

The standard advice to keep the rate window at least four times the scrape interval, and the recording rule interval equal to the scrape interval, follows from this. A recording rule that evaluates faster than the scrape produces duplicate evaluations of identical input; one that evaluates slower decimates the series and moves the aliasing boundary.

Lookback is a staleness rule, not a sampling rule

An instant query, meaning one evaluated at a single timestamp rather than over a range, needs a value for each series at that timestamp. It does not require a sample at exactly that instant. The query engine scans backwards from the evaluation timestamp up to a lookback delta, conventionally five minutes, and uses the most recent sample it finds. Older samples are treated as absent.

This produces a specific behaviour at the edges of a series’ life. When a target disappears, its last sample remains answerable for the lookback period and then the series vanishes from instant queries. That gap is why explicit staleness markers exist: a scrape that succeeds but finds the metric gone can emit a marker that terminates the series immediately rather than letting it linger for five minutes. Without the marker, a target that stops exposing a metric and a target that is still exposing the last value it saw are indistinguishable for the length of the lookback window.

Lookback also interacts with range queries in a way that surprises people. A range query at timestamp $t$ with a range selector of $[5m]$ collects samples in $(t-5m, t]$, and if the lookback delta is also five minutes the two boundaries nearly coincide. When the range is longer than the lookback, the earlier part of the range contributes nothing even though samples exist there, because those samples are stale relative to the evaluation timestamp. The two parameters are independent settings that happen to default to the same value, and changing one without the other changes which samples a query can see.

Why scrape interval changes are visible in the data

Changing the scrape interval of a target changes its apparent behaviour for reasons that have nothing to do with the target. The obvious one is that rate estimates become noisier or smoother. The less obvious one is that the phase of the sampling relative to periodic behaviour in the workload shifts, so an aliased oscillation moves to a different frequency.

A third effect is that the counter’s absolute value is unaffected but the derived increment is not. If a process restarts, the pre-reset value recovered by the correction is the value at the last scrape, and a longer interval means more accumulated traffic is discarded at the reset. The systematic underestimate described earlier scales with the scrape interval, so lengthening the interval silently increases the bias in the long-run rate.

The most common operational mistake is to scrape aggressively because the graphs look better. A fifteen-second interval on a thousand targets is a thousand requests every fifteen seconds, and the collector’s own CPU and network cost is a real constraint on how short the interval can be. The number of samples retained also scales inversely with the interval: at fifteen seconds, a series produces 5,760 samples per day, and a one-year retention of a few million series is a storage problem that the interval directly controls. Cardinality and retention are the other half of the same budget, and Cardinality Explosion and Where It Comes From covers the side of the ledger that dominates in practice.

The interval is also a memory cost

Collection cost is the part of the budget that scales with the inverse of the interval in two independent places. The first is request rate: every target is asked once per period, so halving the period doubles the request count, the connection count, and the parsing work in the collector. The second is the in-memory head of the time-series store, which holds recent samples for every active series so that they can be appended to and queried without touching a compressed block.

The head block’s memory is proportional to the number of active series times the number of samples retained per series, and the retained window is itself usually expressed in time. At a fifteen-second interval, a two-hour head window holds 480 samples per series; at five seconds it holds 1,440. For a few hundred thousand series that difference is the gap between a process that fits in memory and one that does not, and the failure at the boundary is a restart rather than a slowdown. Reading the scrape interval as a pure resolution knob ignores that it is also a multiplier on the most constrained resource in the pipeline.

The same arithmetic governs long-term storage, where the compressed size per sample depends on how smooth the series is rather than on the interval alone. A counter scraped more often produces more samples of smaller increments, and the incremental values are no easier to compress, so halving the interval roughly doubles the stored bytes. The encoding described in Delta-of-Delta Timestamps and XOR Float Encoding on Disk explains why the compression ratio does not improve with a shorter period the way an intuition about “smaller numbers” would suggest.

Resolution is a property of the whole path

The effective resolution of a metric is the coarsest period anywhere between the instrument and the graph. A counter incremented on every request, scraped every fifteen seconds, recorded by a rule every thirty seconds, and queried over a five-minute window has a resolution of five minutes. Optimising the scrape interval while leaving the recording rule at thirty seconds gains nothing for anything that reads the recorded series.

There is no way to recover resolution that was lost. Interpolation between samples produces values that never existed and, for counters, can produce increments that are not integers. The only honest options are to sample faster, which costs collection and storage, or to accept the resolution and stop reading structure into features shorter than it can represent.