Push and Pull Collection Models and Who Owns the Sample Clock

A metric is not a stream of events; it is a reading of a counter or gauge at a point in time. Two collection models disagree about who takes that reading. In the pull model, a collector decides, on its own schedule, to ask each target for its current values. In the push model, the process holding the values decides when to send them. The disagreement sounds procedural. It changes what “the target is gone” means, what happens when a target restarts, and how much work the collector does when the number of targets grows.

The scrape loop as a scheduler

A pull-based collector maintains a list of targets obtained from service discovery, and for each target a scrape loop that fires at a fixed interval with a per-target jitter, then parses the response into samples stamped with the time the request was issued. The timestamp matters: samples are stored at scrape time, not at the time the target computed the value, because the collector cannot observe the latter. The interval is the resolution of the entire dataset, a rate computed over a window shorter than two intervals has too few points to be meaningful, and a counter that increments and is reset within one interval is invisible.

Three details of the loop have outsized consequences.

The first is jitter. If every target were scraped at the same instant, the collector would alternate between idle and saturated, and the storage layer would absorb the whole cluster’s samples in bursts. Distributing scrape offsets across the interval flattens both. The cost is that samples from different targets are not aligned in time, so a query that compares two series at the same instant is comparing readings taken up to one interval apart.

The second is the timeout, which must be strictly less than the interval. A target that hangs rather than refusing the connection occupies a scrape slot until the timeout expires; if the timeout equals the interval, a hung target permanently consumes its slot and the next scrape is already late. Deployments commonly set the timeout to roughly the interval or half of it.

The third is how the response is interpreted. A pull endpoint that reports its own timestamps, as the OpenMetrics exposition format allows, forces the collector to choose between using them and using scrape time. Using the target’s timestamps is more accurate but permits a target to report stale or fabricated times; ignoring them is simpler but discards information. The configuration knob for this choice is one of the few places where the model’s assumptions are exposed directly.

The pull model also imposes a reachability requirement that is easy to underrate: the collector must be able to open a connection to every target. In a network with segmented trust zones, that means the collector sits inside or adjacent to each zone, or a proxy is placed in front of the targets. Push inverts the requirement, targets must reach the collector, which is often the only direction a firewall permits, since egress from an application network to a monitoring endpoint is a smaller concession than ingress to every application host.

Staleness as a substitute for absence

In a push system there is no request to fail, so the collector cannot distinguish “the process is healthy and has nothing new to report” from “the process is dead”. Both look like silence. The standard resolution is a staleness rule: if no sample arrives for a series within some multiple of the expected push interval, the series is marked stale and queries treat it as absent rather than as holding its last value.

The rule is necessary because the alternative is worse. Carrying the last value forward means a crashed process appears to hold a constant value forever, and any alert built on “value has not changed” fires for the wrong reason while an alert on “value is zero” never fires at all. Marking stale converts silence into a well-defined absence, and absence in a query language can be handled deliberately.

Pull systems get the same information for free, and more precisely: a failed scrape is an event with a timestamp, an HTTP status or a connection error, and a duration. The collector records it as an explicit up/down signal alongside the target’s own metrics. This is the strongest argument for pull, and it is not about efficiency at all, it is that failure of the monitoring path and failure of the monitored process are separable observations.

Push systems can recover an equivalent signal only by adding a heartbeat: a synthetic series the process emits on a schedule, whose absence is meaningful. That heartbeat costs one sample per interval per process, which is negligible, and it must be distinguishable from a genuine zero. The heartbeat is the moral equivalent of the scrape’s success bit, and a push deployment without one has no way to tell a quiet service from a dead one.

Restarts, resets, and cumulative temporality

A counter that resets to zero on process restart is the central hazard of counter semantics, and the two models expose it differently.

Under pull, the collector observes the counter at scrape times. A restart between two scrapes shows up as a decrease between consecutive samples, and rate computation treats a decrease as a reset, adding the new value to the previous one as if the counter had wrapped. This works because the collector holds the previous sample itself and can compare.

Under push, the sender may transmit either a cumulative value (the total since process start) or a delta (the increment since the last report). Delta temporality is natural for a push agent that already holds the previous value: it computes the difference locally and sends only the increment, which is smaller and avoids the reset problem entirely at the sender. The cost lands on the receiver, which must associate deltas with the correct series identity and handle a gap, a missed report means a permanently missing increment that no later message repairs.

Cumulative temporality keeps the receiver’s job simple but pushes the reset problem across the network, where the receiver must still detect decreases. Most protocols support both and require the temporality to be declared per stream, because mixing them silently produces sums that are wrong by an arbitrary factor.

The identity question is subtler than the reset question. A series is identified by its name and its label set, and a process that restarts with a new instance identifier produces a new series rather than a reset one. Whether that is desirable depends on whether queries are written against the instance or against the service; the second requires an aggregation that sums across instances, and the aggregation is only correct if stale instances stop contributing. This is exactly the problem that Cardinality Control in Metric Pipelines addresses from the label side.

Short-lived processes and the case push wins

A batch job that runs for ninety seconds cannot be scraped reliably. The collector would have to discover it, schedule a scrape, and hope the process is still alive when the request arrives; a scrape interval of fifteen seconds over a ninety-second lifetime yields a handful of points at best, and a job that finishes between discovery and the first scrape yields none.

Push handles this shape naturally: the job accumulates its metrics in memory and emits them at completion, or emits periodically with a final flush before exit. The receiver sees a short, complete series. The same property makes push the only practical model for functions-as-a-service, where there is no long-lived listener at all and the platform controls process lifetime.

There is a hybrid worth naming: an agent that pulls local endpoints and pushes the result. A node-level agent scrapes the runtime’s own metrics endpoint on the host, an application server’s status page, an exporter bound to loopback, and forwards them to a central collector. The collector never needs network access to the application hosts, discovery stays local, and the failure signal is preserved because the agent knows whether its scrape succeeded. Most large deployments converge on this shape, and it is why the distinction between push and pull is better understood as a question of where the scrape loop lives than as a property of the protocol.

Scale arithmetic in both directions

Pull cost grows with the product of targets and scrape frequency, and it is paid by the collector: one TCP connection or one HTTP request per target per interval. At ten thousand targets and a fifteen-second interval, the collector issues roughly 670 requests per second, and each connection carries a handshake unless keep-alive is used. Connection reuse is therefore not an optimization but a requirement at that size, and the number of concurrent in-flight scrapes must be bounded independently of the target count or a slow batch of targets will exhaust file descriptors.

Push cost grows with the number of reporting processes and their push frequency, but the load is distributed across the senders rather than concentrated in one collector. The concentration is what makes pull easier to reason about: capacity planning is a single number, and a collector that is overloaded is visibly overloaded in its own metrics. Push spreads the load and hides it, so the receiver’s admission control becomes the only place where overload is visible, the subject of Load Shedding and Retry Amplification.

The per-target request rate also decides how much of the pipeline’s work is fixed overhead. Under pull, each scrape is at minimum one request with headers, a connection share, and a response parse, so the cost per target does not shrink with the number of metrics the target exposes. An exposition endpoint reporting two counters and one reporting two hundred cost the collector nearly the same per scrape, which is why the marginal cost of additional instrumentation on a pull-based target is close to zero until the payload itself becomes large.

Neither model is cheaper in general. Pull duplicates work when many collectors scrape the same target, which is why federated or hierarchical pull exists: a mid-tier collector scrapes once and re-exposes an aggregate. Push duplicates nothing but requires every sender to be configured with a destination, and a destination change is a configuration rollout across every process rather than an edit in one place.