Resource Exhaustion Faults: Memory, File Descriptors, and Inodes

Running out of something is the one fault class where the failure is not a discrete event but a gradient. A process that is killed is gone; a process that cannot allocate is told so; a process that is under memory pressure keeps running, more slowly and less predictably, until some threshold converts the pressure into a decision made by the kernel on the process’s behalf. Designing an exhaustion experiment means deciding which point on that gradient is the fault.

Memory: three distinguishable failure modes

The first mode is a refused allocation. When a program asks for memory and the kernel cannot or will not provide it, the request fails with a defined error and the program’s error path runs. This is deterministic, easy to reason about, and comparatively rare on the default Linux configuration, because the kernel’s default heuristic for virtual memory accounting permits overcommit: an allocation reserves address space, and physical pages are assigned on first touch. A large allocation therefore succeeds and the failure is deferred, which is why a program that tests the result of its allocator and assumes success afterwards is not actually protected against running out.

The second mode is pressure, in which the kernel reclaims before it refuses. Reclaimable pages include clean file-backed pages that can be dropped, dirty pages that must be written back first, and anonymous pages that can only be moved to swap if swap exists. Reclaim consumes CPU and I/O, so the process under pressure sees its own throughput fall before anything fails. Writeback thresholds define the point at which dirty data starts being flushed in the background and the harder point at which writers are blocked; the traditional defaults are ratios of available memory rather than absolute values, which is why a container limited to a small fraction of the host behaves quite differently from the host itself.

The third mode is the killer. When reclaim cannot satisfy a request, the kernel selects a victim within the constrained scope. Under a memory cgroup limit the selection is confined to that group, so the blast radius is the workload that was constrained; without a limit the selection is global, and the victim is chosen by a heuristic that combines the process’s memory footprint with an adjustable adjustment value that ranges from strongly protected to strongly disfavoured. A fault injection that intends to test the handling of a killed dependency must account for the fact that the killed process may not be the one the injector targeted.

Swap placement reverses the trade between the first two modes. With no swap, pressure escalates quickly to the killer and the failure is fast and visible. With swap, the same pressure is absorbed as latency while pages move between media, and the system can remain nominally available for much longer while performing far worse. For an experiment, the configuration is part of the fault definition; the same workload under the same injection produces a fast failure or a long degradation depending on it.

Descriptors: per-process and system-wide limits

Open file descriptors are bounded twice, and the two bounds give different errors. The per-process limit is a resource limit with a soft value that the process may raise up to a hard value, and exceeding it fails the call with a per-process error indication. The system-wide bound applies to all descriptors on the host and exceeding it yields a different error, so an experiment that exhausts the host’s supply damages unrelated workloads while an experiment that exhausts one process’s supply does not. Reading the limit, the current usage, and the number of allocated descriptor entries against the global maximum distinguishes which bound is in play.

Descriptor exhaustion is a useful fault because it is invariably a leak test. A workload that holds descriptors correctly survives a repeated fault; one that leaks a descriptor per failed operation accumulates them and eventually fails at a rate that has nothing to do with the injected fault. Injecting the fault repeatedly, rather than once, is what turns this from a pass into a finding, and it is one of the cases where a long-running experiment is more informative than a short one.

Disk: bytes are not the whole story

A filesystem can refuse a write for at least three independent reasons. The first is exhausted space: the count of free blocks reaches zero, or below the reserved fraction that the filesystem keeps for privileged use. Filesystems created with the conventional defaults reserve a few percent of the volume for that purpose, which is why a volume reporting a small amount of free space can still refuse writes from an ordinary user. The second is exhausted inodes: a filesystem has a fixed or bounded number of metadata records, and a directory holding very many small files can reach the limit while a large amount of block space remains unallocated. The third is a quota, which imposes a limit that has no relationship to the underlying free space.

The three are distinguishable from the filesystem statistics interface, which reports total and free blocks, total and free inodes, and the counts available to a non-privileged caller separately. An injector that fills a filesystem with one large file tests space exhaustion; one that creates many empty files tests inode exhaustion; and the failure modes differ, because applications that check free space before writing see plenty of space in the second case and then fail on creation. A filesystem that reaches an unrecoverable error state may also be remounted read-only, which converts a partial failure into a total one, and that transition is itself worth injecting with a device-level fault rather than a fill.

Applying pressure without consuming the host

The modern mechanism for containing these faults is the control group. Memory limits are expressed as a maximum, a high-water mark that triggers reclaim and throttle, and a swap limit; process counts, CPU bandwidth in quota-and-period terms, and I/O rates have their own controls. Writing a limit and then generating load inside the group confines the consequence.

# a cgroup v2 hierarchy: enable the memory controller, then constrain a group
mkdir -p /sys/fs/cgroup/experiment
echo '+memory +pids' > /sys/fs/cgroup/cgroup.subtree_control
echo '200M'          > /sys/fs/cgroup/experiment/memory.max
echo '150M'          > /sys/fs/cgroup/experiment/memory.high   # throttle before killing
echo 64              > /sys/fs/cgroup/experiment/pids.max

# afterwards, an empty value returns the limit to its parent's
echo 'max'           > /sys/fs/cgroup/experiment/memory.max

The distinction between the maximum and the high-water mark is the difference between two experiments. A group that exceeds the high-water mark is throttled and reclaims harder while remaining alive, which produces a slow, sustained degradation; a group that reaches the maximum triggers the group-local killer, which produces a fast failure of a specific process. Testing resilience to degradation and testing recovery from a kill are not the same exercise, and the steady state claimed beforehand, as described in formulating the steady-state hypothesis, will differ between them: one is a throughput claim and the other an availability claim.

Why these faults are harder to abort

Kill and network faults revert by an action: the supervisor restarts, the rule is removed. Exhaustion faults revert by releasing something that was deliberately held, and releasing it is not instantaneous or entirely under the injector’s control. Memory held by a synthetic allocation process is returned when that process exits, but only after the pages are actually freed; under swap pressure the unwinding itself takes time, and the system remains slow throughout. A filled filesystem is cleared by deleting the filler, which costs I/O proportional to what was written, and any process that failed during the experiment may need to be restarted afterwards regardless. Descriptor exhaustion reverts as soon as the holder exits.

There is also a statefulness that makes repeated runs incomparable. The host’s page cache and swap contents at the start of the second run differ from the first, the filesystem’s fragmentation differs, and the reclaim behaviour along the way is therefore different even with an identical limit. Experiments of this class should record the starting state and treat small variations in the outcome as expected rather than as evidence about the system. Faults that are cheap to apply and instant to revert belong in continuous verification, the subject of running fault injection as a regression test; exhaustion belongs in scheduled exercises where the aftermath can be handled deliberately.