Measuring Latency Under Load Without Fooling Yourself

An idle-path round-trip measurement is the easiest number in networking to obtain and one of the least useful. It reports the sum of propagation and serialization on a path where nothing is competing, which is the path’s best case and the case that occurs least often when a user is complaining. The number that predicts user experience is the delay a packet experiences while the path carries the traffic it normally carries, and obtaining that number requires deliberately loading the path and measuring the delay at the same time.

The methodology is not complicated, but it has several ways to go wrong that produce confident and wrong answers. The mistakes cluster around where the probe travels, what generates the load, and how the results are summarised.

Idle delay is a floor, not a prediction

The idle round-trip time of a path is its propagation plus serialization floor, plus whatever processing the endpoints and routers add. It is a useful reference because it bounds what any improvement can achieve, and because the gap between idle and loaded delay is a direct measure of how much queueing the path is adding.

The gap is the quantity of interest. If the idle round trip is 20 milliseconds and the loaded round trip is 220 milliseconds, the path is adding 200 milliseconds of queueing under load, and that 200 milliseconds is what the bufferbloat diagnosis is about. The idle figure alone says nothing about whether that gap exists, and a path with a 20 millisecond idle figure and a 25 millisecond loaded figure is in far better condition than one with a 5 millisecond idle figure and a 300 millisecond loaded figure, even though the second path looks better by the idle number.

Two paths with the same idle figure can differ by an order of magnitude in loaded delay. That is the practical reason the measurement has to include load, and why a comparison between two configurations based on idle pings is not a comparison at all.

Where the probe has to travel

The probe must traverse the bottleneck whose queue is being investigated, and it must not itself become part of the load.

Both conditions are harder to satisfy than they look. A probe sent from the host that is generating the load shares the host’s network stack, its uplink queue, and its CPU with the load generator, so the probe’s own timestamping is delayed by the load. The resulting numbers mix queueing on the path with scheduling delay inside the host, and the second component can dominate when the host is saturated. The standard fix is to run the probe on a separate host or a separate network namespace with its own queue, so that the probe’s packets enter the path independently of the load.

The probe must also travel through the bottleneck rather than around it. On a path with multiple possible routes, or with a load generator attached at a different point than the probe, the probe may take a path that does not include the congested queue. Measuring delay from a data-centre host to a remote target while the load runs between two other hosts tests nothing about the loaded link. The probe’s target must be beyond the bottleneck from the probe’s position, and the load must cross the same bottleneck.

A practical arrangement is a probe host on the local side of the bottleneck sending to a target on the far side, with the load generated between the same two sides. The probe’s packets then queue behind the load at exactly the point of interest.

Load patterns that expose the queue

The load’s shape determines what the measurement finds. Three patterns are useful, and they answer different questions.

Saturating load: enough parallel flows to fill the link completely, run for long enough that the queue reaches its steady state. This exposes the maximum standing queue and gives the worst-case delay a user could see. It is the pattern that reveals bufferbloat most clearly, because a large buffer under saturating load settles at a deep standing queue that persists for the run’s duration.

Ramp load: a gradually increasing offered rate, from well below capacity to above it. This shows where the delay curve bends, which is the empirical counterpart of the utilisation knee. The bend’s position identifies the utilisation at which the path starts adding delay, and comparing it with the path’s rated capacity indicates how much headroom is actually usable for latency-sensitive traffic.

Bursty load: short, intense bursts separated by idle intervals. This tests whether the queue recovers between bursts or accumulates. A path whose queue drains fully between bursts adds delay only during the bursts; a path whose queue does not drain has a standing queue that persists. The distinction separates a buffer that is appropriately sized from one that is too large for the traffic pattern.

The measurement has to sample the delay continuously during each pattern, not before and after. A single probe before the load and a single probe after it measures the idle path twice. The samples taken during the load are the data; the samples before and after establish the floor and confirm that the path returns to it.

Summarising a distribution honestly

Delay under load is a distribution with a long right tail, and summarising it with a mean discards the part that matters. The mean is dominated by the bulk of samples that queue moderately, while the samples that determine whether a voice call is usable are the ones in the upper percentiles.

Reporting a median, a high percentile, and a maximum is more informative than a mean. The median describes the typical experience, the high percentile describes what happens often enough to be noticed, and the maximum describes the worst case the buffer permits. A path whose median is 30 milliseconds and whose ninety-ninth percentile is 400 milliseconds is qualitatively different from one whose median is 60 and whose ninety-ninth percentile is 80, even if their means are close.

The sample count has to be large enough for the percentile to mean something. A ninety-ninth percentile estimated from a hundred samples is the second-worst sample and carries almost no information; a few thousand samples give a percentile that is stable enough to compare between runs. The samples are also correlated, because consecutive probes experience the same queue, so the effective sample size is smaller than the count suggests. Spacing the probes by more than the queue’s drain time reduces the correlation, at the cost of needing a longer run.

// Sample round-trip times at a fixed interval while the load runs elsewhere.
func sample(ctx context.Context, target string, interval time.Duration) (rtts []time.Duration, losses int) {
	tick := time.NewTicker(interval)
	defer tick.Stop()
	for {
		select {
		case <-ctx.Done():
			return rtts, losses
		case <-tick.C:
			start := time.Now()
			conn, err := net.DialTimeout("tcp", target, 2*time.Second)
			if err != nil {
				losses++ // counted separately: a timeout is not a zero RTT
				continue
			}
			conn.Close()
			rtts = append(rtts, time.Since(start))
		}
	}
}

// pct returns the p-th percentile of d, with d reordered in place.
func pct(d []time.Duration, p float64) time.Duration {
	sort.Slice(d, func(i, j int) bool { return d[i] < d[j] })
	idx := int(p / 100 * float64(len(d)-1))
	return d[idx]
}

Dialling and closing a TCP connection measures one round trip and no more, and it needs no cooperation from the target beyond a listening port. The error branch is the important part: a probe that timed out is counted as a loss rather than appended to the delay samples, because dropping it silently removes exactly the samples that would have raised the high percentiles.

Loss has to be reported alongside delay. A path at very high utilisation shows a moderate mean delay and a substantial loss rate, because the packets that would have waited longest were discarded rather than delayed. Reporting delay without loss makes such a path look better than it is. The IPPM metrics define one-way delay and one-way loss as separate quantities for exactly this reason, and a measurement that reports only one of them is incomplete.

Reproducing and comparing runs

Two runs of a delay-under-load measurement are comparable only if the load, the path, and the probe placement are the same. Changing any of the three changes the number, and the change may be larger than the effect being investigated.

The load’s rate must be controlled and reported, not merely described as “a large transfer”. A transfer that achieves 40 Mbit/s on one path and 400 on another offers very different pressure to their respective bottlenecks. The load’s flow count matters too: a single flow’s congestion control determines its own queueing behaviour, and several concurrent flows produce a different queue than one flow at the same aggregate rate.

The path must be stable across the runs. A route change during a measurement replaces the object being measured, and the delay distribution changes for reasons unrelated to the configuration change under test. Recording the path, or at least the hop count and the addresses of the first few hops, makes it possible to notice that the comparison was invalid.

Probe placement must be identical, including whether the probe shares a host with the load. A comparison between a run with a co-located probe and a run with a dedicated probe measures the difference in methodology, not the difference in configuration.

What the numbers can support

A delay-under-load measurement supports statements of the form “under this load pattern, this path added this much queueing delay at this percentile”. It does not support statements about typical user experience, because that depends on the load patterns users actually produce, which vary by application, time of day, and number of devices. It also does not attribute the delay to a specific device unless the measurement is repeated at several points along the path, which requires probes at each point and is a separate exercise.

What it does support is comparison. Two configurations of the same path measured the same way, or the same configuration measured before and after a change, give a difference that is meaningful if it exceeds the run-to-run variation. Establishing that variation requires repeating the run several times with nothing changed, and it is usually larger than intuition suggests, because the load generator’s own behaviour varies between runs.

The result that most often surprises is the size of the gap between idle and loaded delay on paths that look healthy by every other measure. A link can be delivering its full rated throughput, losing nothing, and adding several hundred milliseconds of delay, all at once, and only a measurement that loads the path and probes through it will show the third fact. That is the case the methodology exists to find, and the mechanism behind it is the standing queue described in standing queues and buffer sizing, with the remedies in active queue management and latency under load.