DevOps · Cluster
Chaos and Drills
Fault injection and game days.
- 01 Steady-State Hypotheses and the Abort Condition in Fault Injection How a fault-injection exercise defines normal behaviour as a falsifiable claim about metrics, how tolerance windows set its resolution, and why the abort condition is a separate signal.
- 02 Bounding Blast Radius: How a Fault Injection Chooses Its Targets The three axes that bound an injected fault, how deterministic partitioning selects a stable subset of targets, and why aggregates that span treated and untreated hosts hide the result.
- 03 What a Kill Fault Looks Like to the Kernel: Signals, Freezes, and Orphans The difference between SIGTERM, SIGKILL, and freezing a process, what each leaves behind in sockets and locks, and why reaping behaviour decides whether a container restarts cleanly.
- 04 Packet-Level Fault Injection: Latency, Loss, and Partitioning How delay, loss, and reordering are emulated in the kernel queueing discipline, why DROP and REJECT produce different failures in callers, and how TCP converts a small packet loss into a long stall.
- 05 Clock Skew Injection and the Failures That Depend on Wall Time The difference between clock offset, skew, and drift, the two ways to inject a time fault without breaking the host, and which classes of software fail when wall time jumps.
- 06 Injecting Failures at the RPC Boundary: Codes, Timeouts, and Resets Why the shape of an injected failure decides the caller behaviour it provokes, how deadlines and retries multiply load, and what circuit breaker states actually reveal under injection.
- 07 Resource Exhaustion Faults: Memory, File Descriptors, and Inodes How memory pressure differs from allocation failure, why descriptor and inode exhaustion produce distinct error codes, and how control groups let a fault be applied without touching the host.
- 08 Structuring a Game Day: Scenario, Roles, and Safety Nets What separates a drill from an experiment, how a scenario document is written so that it constrains the fault and not the response, and the safety nets that must exist before the exercise starts.
- 09 Separating Injected Faults from Organic Failures in Telemetry How to mark injected traffic so it survives every hop, why filtering at query time beats deleting at ingestion, and how to keep an exercise out of the alert path without blinding it.
- 10 Fault Injection as a Continuous Regression Test How an experiment becomes a repeatable test case with a pass predicate, why resilience tests fail for environmental reasons far more often than for real regressions, and what a green suite does not prove.