Flow Sampling Error When Deciding Whether a Link Is Full

Interface counters count every byte and every frame, and they answer the question of whether a link is full exactly. Flow records answer the more useful question of what the link is full of, and they answer it approximately, with an error that is a function of the sampling rate and of the size of each conversation. The two measurements are not interchangeable, and a capacity decision made from flow data alone inherits an error that can be larger than the change being investigated.

Two measurement models that are often conflated

The first model samples packets. A device selects one packet in every N, by a counter or by a random draw, and exports a record describing that packet, including its headers and its length. Volume is estimated by multiplying each sampled packet’s length by N and summing. This is the model the packet sampling specifications describe, and it is the model that most switch chips implement, because it requires no state per flow.

The second model meters flows. The device maintains a table keyed by the fields that define a flow, counts packets and bytes for each entry, and exports a record when the entry is removed or when a periodic timer expires. Sampling may be applied on top, but the counting within an entry is exact for the packets the device saw. This is the model the flow export protocol describes, and it produces per-flow totals with no per-packet multiplication.

The distinction matters because the error behaves differently. Packet sampling has an error that depends on how many packets of a conversation were sampled, so it degrades as conversations get smaller. Flow metering has an exact count per entry, and its errors come from the table’s eviction policy and from the timers that close entries, so it degrades for long conversations and for tables under pressure.

The arithmetic of a sampled volume estimate

For uniform sampling at rate one in N, the estimator for a flow’s byte total is the sum of the sampled lengths multiplied by N. The estimator is unbiased: its expected value equals the true total. Its variance depends on how many packets were drawn, and for a flow of many independent packets the relative standard error is approximately one over the square root of the number of samples.

That single relation governs everything that follows. A conversation that produced a thousand sampled packets has a volume estimate with a relative error around three percent. A conversation that produced ten sampled packets has one around thirty percent. A conversation that produced one sampled packet has an error of order one hundred percent, and a conversation that produced none is not in the data at all, with an implied volume of zero.

The number of samples, not the volume, is what controls the error, so the error is worst exactly where the traffic is most numerous and least significant. A host scanning many addresses, a DNS resolver answering short queries, a monitoring system probing every endpoint: each produces a large number of tiny conversations, each contributing few samples, and each poorly estimated. Their aggregate is well estimated, because the aggregate draws on many samples. The individual records are not.

What disappears from the data

A conversation below the sampling threshold is invisible. At a rate of one in a thousand, a conversation of a hundred packets has roughly a one in ten chance of appearing at all. If it does appear, it appears as one or two packets whose lengths, multiplied by a thousand, produce an estimate that is either zero or wildly wrong.

This is the mechanism behind a familiar disagreement: flow data shows no traffic from a host that is known to be sending, or shows traffic from a host at a volume that cannot be reconciled with anything. Neither observation is an error in the exporter. Both are consequences of a sampling rate chosen to keep the export volume manageable, applied to a conversation that is small relative to that rate.

The practical implication is that flow data is a census of the large, not a list of the present. Any question of the form “did this host send anything” is answered badly by sampled flow data and answered exactly by a counter, a log, or a capture filtered to that host. Questions of the form “what fraction of the link is this application” are answered well, provided the application’s traffic is large enough to be sampled many times per interval.

Timers reshape the records without changing the traffic

Flow metering closes an entry for one of two reasons. An inactive timeout closes an entry that has received no packets for a configured interval. An active timeout closes an entry that has been open for a configured interval, regardless of activity. Both exist because the table is finite and because a consumer wants records at a bounded delay.

The active timeout is the one that changes what the data appears to say. A long-lived conversation, such as a bulk transfer or a tunnel carrying many inner conversations, is split into one record per active interval. Each record is exact for its interval, and the sequence of records describes a rate over time, which is useful. But the number of records is now a function of the timeout and the conversation’s duration, not of the conversation count, so counting records as a proxy for counting conversations inflates the count by the duration divided by the timeout.

The inactive timeout interacts with the table size. If the table is full, entries are evicted before their timeout, and the eviction exports a record. Whether the record carries the entry’s accumulated counts or is discarded depends on the implementation, and the difference is a silent loss of traffic in the totals. Comparing the flow-derived total against the interface counter over the same interval is the only way to detect that loss, and the comparison should be routine rather than exceptional.

Sampling both directions double counts

A device that samples on ingress and on egress of the same interface, or on both interfaces of a transit path, produces two records for each packet that crosses both points. The records are distinct and both are correct; summing them produces twice the traffic.

The error is easy to make because the two records look different. Ingress sampling records the packet before any rewrite the device performs, and egress records it after, so the addresses, the time-to-live, and the length may differ. A consumer that keys records by the tuple as recorded will see two conversations rather than one, and will attribute half the volume to each, which is a subtler error than doubling the total: the per-conversation figures are now wrong by a factor of two in a way that no total reveals.

The rule that avoids this is to designate one observation point per interface and to state it in the metadata of any derived dataset. Where both are needed, the records must be separated by direction before aggregation, and the direction must come from the exporter’s own metadata rather than from inference about the headers. Records from two observation points also carry timestamps from two clocks, which is the same problem that makes a merged event timeline uncertain, and correlating logs across devices sets out the bound that applies.

Distinguishing a real change from sampling noise

The decision that sampling error most often corrupts is the comparison of two intervals. Traffic in one direction rose from one interval to the next, and the question is whether the rise is real.

Three checks make the comparison sound. First, compare against the interface counter over the same interval, which is exact: if the counter shows a rise of the same magnitude, the change is real regardless of what the flow data says about its composition. Second, check the sample counts behind the comparison. A change in a category whose sample count is in the tens is within the noise of the estimator, and reporting it as a trend is reporting sampling variance.

Third, check whether the sampling rate itself changed. Exporters that adjust the rate under load, or that are reconfigured, produce records whose per-packet multiplier differs from the one assumed by the consumer. The exporter reports the rate in the record’s metadata precisely so this can be handled, and a consumer that ignores it produces volume estimates that are wrong by the ratio of the rates, in a way that looks like a traffic change of exactly that ratio.

def volume(sampled_bytes, n_samples, rate, z=1.96):
    """Point estimate and interval for a uniformly sampled flow volume."""
    est = sampled_bytes * rate
    # relative standard error ~ 1/sqrt(n) for independent packet draws
    rel = z / n_samples ** 0.5 if n_samples else float("inf")
    return est, (est * (1 - rel), est * (1 + rel))

The interval is the honest output, and it is often wider than the effect being claimed. A capacity decision that survives an interval of plus or minus thirty percent is a decision the data supports; one that does not is a decision that needs a counter, not a better chart.

Where flow data is the right instrument

Sampling error is a property of the question, not a defect of the instrument. Flow data is the right instrument when the question is about composition at a scale where each category is sampled many times per interval, when the answer tolerates an interval of a few percent, and when per-conversation attribution matters more than exactness. It is the wrong instrument for detecting the presence of small conversations, for measuring a single conversation’s throughput, and for deciding whether a link is at its limit, which the interface counters answer exactly.

The boundary between the two instruments is where the diagnosis of a saturated link becomes difficult, because the exact measurement says the link is full and the approximate one says which conversations fill it, and the two disagree at the margin. Resolving that disagreement means observing the packets themselves at the point of congestion, with the artifacts that a capture at that point introduces, as described in packet capture placement.