Networking · Network Operations

Where a Packet Capture Sits Determines What It Shows

How capture points on hosts, bridges, and taps differ in what they observe, and why segmentation offload and checksum offload make a local capture disagree with the wire.

A capture is a copy of packets at one point in a path, and every question it can answer is bounded by that point. A capture taken inside a host’s kernel sees what the kernel decided to transmit; a capture taken on a switch port sees what the wire carried. When the two disagree, the disagreement is usually not evidence of loss but evidence that the host’s network stack was doing something the capture point could not see. Distinguishing the two requires knowing which artifacts each point produces.

The capture points and what each one can witness

The most common point is a host’s own interface in promiscuous mode, which receives copies of frames the driver would otherwise discard. For outgoing traffic the copy is taken after the protocol stack has handed a buffer to the device, so it reflects the stack’s view: what it chose to send, with the headers it built.

A bridge or virtual switch inside a host offers a second point, and it sits at a different layer of the same machine. Traffic between two virtual machines on the same bridge may never reach a physical interface at all, so a capture on the physical interface shows nothing while the bridge shows everything. This is the standard explanation for traffic that a monitoring host cannot see: the flows are local to the virtual bridge and never traverse the monitored link.

A physical tap or a switch port with port mirroring offers a third point, outside the host. It sees the frames that actually crossed the medium, including any the host’s stack generated but the driver later discarded, and excluding any that the host’s stack processed without transmitting. A mirror also has its own failure modes: a mirror configured to copy only ingress or only egress shows a half-duplex view of a conversation, and a mirror on a congested switch may drop copies under load while the original traffic passes untouched.

The consequence for diagnosis is that a missing packet is only meaningful relative to the point. A capture at the host showing a segment that never appears at the mirror is strong evidence of a transmit-side drop. The reverse, a segment at the mirror with no counterpart in the host capture, is not evidence of anything except that the capture filter or the buffer was too small.

Segmentation offload rewrites the packet after the capture point

Modern network adapters accept buffers far larger than the maximum transmission unit and split them into wire-sized frames themselves. The host stack builds one large buffer with a single set of headers, marks it for segmentation, and hands it to the driver. A capture taken above the driver records that large buffer: a single “packet” whose length exceeds the MTU, carrying one checksum field that has not yet been computed per segment.

This is why a capture on a sending host routinely shows packets of tens of kilobytes on an Ethernet link whose MTU is 1500 bytes. The capture is accurate about what the stack produced and wrong about what the wire carried. The frame count is also wrong, since one captured buffer becomes many frames, and any analysis that counts packets or computes retransmission rates from such a capture is measuring the stack’s batching rather than the network’s behaviour.

The receive side has a mirror-image artifact. Receive coalescing merges several arriving frames into one buffer before the stack processes them, and a capture taken after coalescing shows fewer, larger packets than the wire carried. Where the capture sits relative to coalescing determines whether it sees individual frames or merged ones, and on a busy interface the difference can be an order of magnitude in packet count.

The standard remedy is to disable the offloads for the duration of the capture, or to capture at a point below them. Turning off segmentation offload on a sending host makes the stack segment in software, so the capture matches the wire at the cost of additional CPU work per packet. Neither approach changes what a mirror sees, because the mirror is already below the adapter’s segmentation.

ethtool -k eth0                  # current offload state: tso, gso, tx checksumming
tcpdump -i eth0 -s 0 -w cap.pcap  # full-length capture, no truncation

Reading the offload state before the capture, and recording it beside the file, is what makes a later argument about a missing packet resolvable. A capture file whose offload state is unknown cannot distinguish a segmentation artifact from a real difference in what was sent.

Checksum fields that are not yet valid

Checksum offload produces a second class of artifact that is more misleading than the size mismatch. The stack leaves the transport checksum field in a state that the adapter will complete during transmission, and that state is not the correct checksum of the packet as captured. An analyser that verifies checksums will therefore flag every outgoing packet from an offloading host as corrupt.

The field often contains a recognizable partial value rather than a plausible checksum, and its exact contents depend on the driver’s convention, so there is no general rule for interpreting it. The correct reading is that checksum validation is only meaningful at a point where no offload is in effect. Captures taken at a mirror, or on a host with checksum offload disabled, carry complete checksums; captures taken above an offloading driver do not.

Receive-side checksum offload creates the opposite trap. The adapter may report that it verified a checksum without the stack ever computing it, so the stack accepts a segment whose checksum is wrong. A capture on the receiving host shows the segment as received and accepted, and only a capture on the wire, with verification enabled, shows that the checksum was bad. This matters because a corrupt segment that passes an unverified path can produce an application-level symptom with no transport-level error anywhere in the logs.

Timestamps and the illusion of ordering

Capture timestamps are taken when the kernel or the adapter delivers the copy, not when the frame crossed the medium. Under load the delivery can be delayed by the time it takes to drain a queue, so two packets that arrived microseconds apart can be timestamped milliseconds apart, in an order that reflects queueing rather than arrival.

The effect is most visible when comparing captures from two points on the same path. One-way delay computed from the difference between a timestamp at the sender and a timestamp at the receiver includes the capture latency at both points, which is unbounded and load-dependent. A measured delay that grows with traffic volume may be queueing in the network or queueing in the capture path, and the two are not distinguishable from the timestamps alone.

Hardware timestamping, where the adapter records the time at which the frame was received, removes most of this uncertainty on the receive side. It requires support in the adapter and the driver, and it is not available on every platform, so the practical rule is that timestamps are suitable for ordering packets within a single capture and for coarse interval measurement, not for delay measurement across two hosts.

Buffer and filter effects that look like network loss

A capture that cannot keep up silently drops. The kernel’s capture buffer is finite, the default snapshot length may truncate large frames, and a filter that is too permissive pushes more work into the capture path. When the buffer overflows, the capture tool reports the number of dropped packets, and that number is the only reliable indicator that the capture is incomplete. A capture without a drop count is not evidence that nothing was lost.

Snapshot length deserves separate attention because its effect is subtle. Truncating to a small value keeps only the link and network headers, which is sufficient for addressing questions and insufficient for anything that needs payload or options. A truncated capture also changes the apparent packet length, so a length-based analysis of such a capture measures the snapshot setting.

Filters are evaluated against the packet as presented at the capture point, which reintroduces the offload problem. A filter that matches on a transport port sees the port in the large pre-segmentation buffer, so a filter for a specific port catches the buffer once rather than each segment. Filtering on a field that the adapter would rewrite, such as a checksum, is meaningless at that point.

Choosing the point from the symptom

The symptom determines the point. A failure that affects one host’s ability to reach anything suggests the host’s own stack and the first hop, so a host capture plus a neighbour cache inspection covers most cases; the sequence for that is described in interface configuration order. A failure that affects one conversation between two specific hosts suggests the path, so captures at both endpoints with a common clock reference and, if possible, a mirror on the intermediate device are needed.

A failure that affects everything on one link, but not the link’s own traffic counters, points at filtering or at a device that is dropping silently. The counters that report those drops, and the difference between a drop the interface attributes to a queue and one it attributes to a policy, are the subject of interface drop counters.

The one artifact that survives all of these is the record of what the capture point actually was. A capture file without a note of the interface, the offload state, the snapshot length, and the drop count cannot be re-interpreted later, and most disagreements about what a capture proves come from that missing context rather than from the packets.

References

  1. pcap-filter(7) — Linux manual page — The Linux man-pages project
  2. tcpdump(8) — Linux manual page — The Linux man-pages project
  3. RFC 9293: Transmission Control Protocol (TCP) — IETF