Packet-Level Fault Injection: Latency, Loss, and Partitioning

Network faults are attractive because they sit between the two failure shapes that are easy to reason about: a process that is gone, and a process that is answering. A network fault keeps the process alive while making its traffic arbitrary, and the caller’s experience then depends on decisions made far below the application, in the retransmission behaviour of the transport protocol and in the queueing discipline that shapes the packets.

The queueing discipline as an injection point

On Linux, the traffic-control subsystem attaches a queueing discipline to a network interface, and packets leaving that interface pass through it. The network emulator discipline implements the faults that are interesting for experiments: delay with optional jitter and correlation, packet loss, duplication, reordering, and rate limitation. A discipline is applied to one interface and can be made selective by attaching filters, so a fault can target a destination address, a port, or a traffic class without touching other traffic.

# 200 ms base delay, plus or minus 50 ms, correlated by 25 percent
tc qdisc add dev eth0 root netem delay 200ms 50ms 25%

# then mutate the same discipline: 5 percent loss and 1 percent duplication
tc qdisc change dev eth0 root netem delay 200ms 50ms 25% loss 5% duplicate 1%

Two structural facts about this mechanism shape what can be tested. First, delay is imposed on the egress path of the interface it is attached to, so shaping both directions of a conversation requires a discipline on both ends or on both interfaces of an intermediate node. Second, jitter with correlation produces bursts of similarly delayed packets rather than independent samples, which is closer to real congestion and considerably more disruptive to real-time streams than uniform jitter of the same amplitude.

Selective loss is the sharpest tool. A discipline can drop packets probabilistically, which degrades throughput smoothly, or a filter can drop packets from one flow entirely, which produces the complete silence of a partition. Those two faults look similar in a graph of request latency and are entirely different in a graph of connection state, because a probabilistic drop leaves the connection alive and a filter-based partition eventually kills it.

Partitioning: what the caller observes

The second family of network faults is at the connection level, and here the mechanism choice determines the observable failure. Discarding packets silently produces a black hole: no response arrives, and the caller learns of the failure only when its own timeout expires. Rejecting with an error response produces immediate failure: the connection attempt is refused and the caller’s error-handling path runs within a round trip. Denying name resolution produces a third shape, in which the failure appears during connection setup and often looks like a configuration error rather than a network fault.

Which of these is correct depends on what is being tested. Blackholing tests timeouts, deadlines, and the distinction between a slow dependency and a dead one; it is the fault that finds clients with no timeout at all, because they hang indefinitely. Refusal tests failover speed, because it is the fastest signal a caller can receive. There is no fault type that covers both, and an experiment that injects only one of them systematically misses the bugs the other finds.

Partitioning a replica also has to be distinguished from killing it, which is the point where transport behaviour and replication behaviour interlock. A partitioned node in a consensus group still believes it is a member; it keeps trying to participate, and if it can reach a minority of peers it may campaign repeatedly for leadership. A transient partition therefore produces a different disturbance pattern from a clean removal, and the number of nodes that remain mutually reachable decides whether the cluster makes progress at all: with 2f+1 members, any partition that leaves fewer than f+1 nodes on the larger side cannot commit. A partitioned node is nonetheless a running one: its process is alive, its threads are scheduled, and it is only its traffic that has been diverted. The distinction between a partition and a freeze decides what the peer observes, because a frozen process sends nothing and closes nothing, while a filtered process sends into a discard and can still be reached on paths the filter does not cover. The kernel’s view of kill and freeze faults is therefore the right reference point when deciding which of the two better reproduces a real failure. The partition is a fault on the topology, and its consequences are computed by the replication protocol, not by the transport.

How packet loss becomes a stall

The most counterintuitive result of running these faults is how little loss is needed to produce a large latency increase, and the reason is the retransmission timer. When transmitted data is not acknowledged within a timeout, the sender retransmits; that timer is computed from smoothed measurements of the round-trip time, and the specification for computing it sets a minimum value so that spurious timeouts on short paths are avoided. Losing a single segment from a stream therefore does not cost one round trip; it costs roughly the retransmission timeout, which is a full order of magnitude larger than the round trip on a local network.

Successive losses back off exponentially, so a short burst of drops can leave a connection waiting several timeout intervals before it recovers. Congestion control then compounds this: most deployed implementations treat loss as a congestion signal and reduce the amount of data in flight, so the connection re-enters slow start or additive increase after the retransmission succeeds. The window takes many round trips to climb back, and a client measuring per-request latency sees a long tail made of a few unlucky requests rather than a uniform slowdown. This is why tail latency is the correct hypothesis for a loss experiment and mean latency is usually not: the mean moves by a small amount while the 99th percentile moves by a large one.

Duplication and reordering produce the opposite bias, and they are the faults that find assumptions about idempotency. A duplicated request that reaches an application without request-level deduplication is processed twice, which is invisible to the transport layer and potentially visible in a ledger. Reordering stresses reassembly buffers and any application-level assumption that message order follows send order. Both are cheap to inject at the packet layer and essentially impossible to inject above it.

Rate limiting and asymmetric faults

Rate limits, expressed as a token bucket or a packet rate, test a different property than loss: they test behaviour under sustained compression rather than under discrete failure. A service that handles a hundred connections per second and is limited to ten does not fail, it queues, and the queue is where the interesting behaviour lives, since queue growth eventually converts a latency problem into a memory problem and then into the resource exhaustion faults of memory, descriptor, and disk pressure.

A rate limit attached to a whole interface also has no notion of which connections matter, so it compresses the traffic of unrelated callers along with the intended targets; rate limits are one of the fault types whose containment is decided by placement rather than by parameters, in the sense described in bounding the blast radius of a fault. Asymmetry is the subtle axis. Injecting delay on the egress path of the server affects responses but not requests; injecting it on the client affects requests and therefore the point in the exchange at which the deadline is first tested. Since a request usually has a shorter budget than a response, the two placements detect different bugs. Likewise, a fault applied only to one port tests whether the system’s health checks and its data path fail together or separately, which is the failure that turns a partial outage into a full one when the checker is on the unaffected port. Choosing the placement deliberately, and recording it with the result, is what makes a network experiment reproducible rather than anecdotal.