Networking · Network Performance

What ICMP Echo and TTL Probing Actually Measure

The difference between the forward and return paths a traceroute samples, why hop times are often wrong, and what rate limiting on the control plane does to the numbers.

An echo request sent to a remote host and answered returns a number that is a round trip over two paths that need not be the same length, through a device whose reply may be generated at a lower priority than its forwarding, and measured by a clock that may be adjusted between the send and the receive. Each of those caveats is a source of error large enough to matter, and none of them is visible in the single number a ping prints.

The second tool, path probing by time-to-live expiry, adds a further layer. It infers topology from error messages, and those error messages are generated by routers under conditions that differ from the conditions under which they forward traffic.

The echo exchange and its assumptions

An echo request is an ICMP message of type 8, and the reply is type 0. The payload is echoed verbatim, and it usually carries a timestamp and a sequence number so that the sender can match replies to requests and compute a round-trip time.

The measurement is a round-trip time by construction. The sender records the time it transmits, the receiver’s kernel replies, and the sender records the time the reply arrives. The difference includes the forward path, the time the target spent handling the request, and the return path. If the forward and reverse paths have different lengths, the reported figure is their sum and not twice either one. On the public Internet asymmetry is common: routing policy, traffic engineering, and the simple fact that the two directions are separate decisions made by different networks all produce paths that differ by tens of milliseconds on intercontinental routes.

The target’s processing time is included and is usually small, but not always. A host that is heavily loaded may take milliseconds to answer an echo request that a router would forward in microseconds. A host that rate-limits ICMP may answer some requests and drop others, which produces loss that has nothing to do with the network. The measured time is therefore an upper bound on the path delay, not an estimate of it.

There is a further subtlety in what a ping does not measure. It does not measure the delay experienced by a TCP connection, because the queueing that a TCP flow sees depends on its own packet sizes and on the treatment its packets receive relative to other traffic. A ping of 20 ms on an idle path says nothing about the delay a bulk transfer would see on the same path under load. That difference is the reason measuring latency under load exists as a separate discipline.

Time-to-live expiry as a topology probe

Every IPv4 packet carries an eight-bit time-to-live field. Each router that forwards the packet decrements it; a router that receives a packet with a value of one discards it and, under the rules in RFC 1812, sends back an ICMP time exceeded message, type 11 code 0. The source address of that error message is the router’s address on the interface it used to send the reply, which is usually but not always the interface the original packet arrived on.

A path probe exploits this by sending packets with increasing time-to-live values. A packet with time-to-live 1 is discarded by the first router, which returns an error naming itself. A packet with time-to-live 2 is discarded by the second, and so on. The sequence of error messages enumerates the routers along the forward path, and the time between sending each packet and receiving its error gives a round trip from the sender to that hop and back.

Two properties of this method deserve emphasis.

First, the measurement is to the hop, not through it. The round trip measured for hop $n$ is the time to reach hop $n$ plus the time for hop $n$ to generate and return an error. It is not the time to reach hop $n+1$. The difference between consecutive hop times is therefore not the delay of the intervening link; it is the difference between two round trips, each of which includes a different device’s error-generation path.

Second, the return path from hop $n$ to the sender need not be the reverse of the forward path to hop $n$. The error message is routed independently, so the reported time mixes a forward path to the hop with a return path from the hop, and neither is necessarily the path that data traffic takes.

The set of addresses returned can also be incomplete or misleading. A router may be configured not to send time exceeded messages at all, in which case the hop shows as a timeout and the probe cannot distinguish “this router is silent” from “this router is unreachable”. A router may answer from a different interface address than the one the probe traversed, which makes the same physical device appear as two hops in successive probes. A router may apply a policy that generates the error with a low priority, so the measured time reflects the control-plane queue rather than the forwarding path.

Rate limiting and the numbers it produces

Routers commonly rate-limit the generation of ICMP error messages and echo replies, because generating them is work that competes with forwarding and because unrestricted generation is a denial-of-service amplifier. The limits are usually implemented as a token bucket: a small burst allowance and a sustained rate of a few messages per second.

The effect on a path probe is systematic. The first few probes of a run get answered promptly, and later ones, especially when several hops are probed in quick succession, find the bucket empty and wait for it to refill. The measured time for those hops includes the wait, which can be hundreds of milliseconds or a full second, and it is entirely an artifact of the router’s politeness policy. A hop that appears to have 300 ms of latency relative to its neighbours may have 1 ms of forwarding latency and a rate limiter.

Echo replies to a target host are subject to the same treatment on many systems, so a ping run that shows intermittent loss and wildly varying times may be measuring the target’s ICMP rate limiter rather than the network. The diagnostic is to compare with a measurement that does not depend on ICMP, such as a TCP connection’s handshake time or an application-level timestamp exchange.

Clock and path effects on the reported figure

A round-trip measurement cancels the sender’s own clock offset, because the same clock is read at both ends. It does not cancel a clock adjustment that happens during the measurement. If the system clock is stepped or slewed between the transmit and receive timestamps, the reported interval includes the adjustment. On systems that synchronise their clocks continuously, a small slew rate is always present, and on systems that step the clock on synchronisation events the error can be large and one-sided.

The other systematic error is the path’s own variation. A single echo request measures one sample of a distribution that includes queueing. Ten requests sent back to back measure ten samples from the same instant of congestion, not ten independent samples of the path’s typical behaviour. The samples are correlated, so their mean is not a reliable estimate of the mean delay, and their minimum is a better estimate of the propagation-plus-serialization floor. Taking the minimum over a run is a standard technique for exactly this reason: the minimum is the sample least contaminated by queueing, and it approximates the path’s fixed delay.

What the tools are good for

Echo and time-to-live probing are good at answering coarse questions. They establish reachability, they give an order-of-magnitude round-trip figure, and they enumerate the routers on a forward path in a simple network where nothing rate-limits and nothing is asymmetric. They are the first tool to reach for when a path is suspected of being broken entirely.

They are poor at answering the questions that latency work actually asks. They cannot separate the forward and return paths, they cannot measure the delay a specific flow’s packets experience, they cannot attribute delay to queueing versus distance, and their hop-by-hop numbers are contaminated by error-generation policies. For those questions the appropriate instruments are timestamped probes at known observation points, one-way delay measurements with synchronised clocks, and passive observation of real traffic. The one-way case is treated in one-way delay and the clock offset problem, and the question of which term of the delay budget a given probe is actually sampling is taken up in decomposing path latency.

References

  1. RFC 792: Internet Control Message Protocol — IETF
  2. RFC 1812: Requirements for IP Version 4 Routers — IETF
  3. RFC 1393: Traceroute Using an IP Option — IETF