What a Kill Fault Looks Like to the Kernel: Signals, Freezes, and Orphans

Killing a process is the cheapest fault to inject and the one whose semantics are most often assumed rather than known. The three common variants — polite termination, immediate termination, and suspension — produce different states in the kernel, are observable by different recovery mechanisms, and leave different residue behind. An experiment that intends to test “the service survives losing a replica” may in fact be testing “the service survives an orderly shutdown”, which is a much weaker property.

Termination signals and the handler window

A termination signal delivered to a process is asynchronous with respect to its execution, and the delivery model is what creates the handler window. SIGTERM has a default action of terminating the process, but a handler can override that: on receipt, the process is interrupted at some convenient point, the handler runs, and if the handler returns without exiting, the process continues. Everything between signal delivery and the process’s final exit is application-defined, and it is where graceful shutdown lives: draining in-flight requests, finishing a transaction, deregistering from service discovery, flushing buffers.

SIGKILL has no such window. It cannot be caught, blocked, or ignored, and the process is destroyed at the next opportunity rather than at a point of the application’s choosing. The same is true of SIGSTOP, which suspends execution without terminating, and it is why a freeze is a strictly different fault from a kill. A frozen process holds every resource it held at the moment of suspension — memory, open sockets, held locks, established connections — while consuming no CPU. Its peers see a node that is unreachable rather than one that is gone, which is precisely the failure mode that distinguishes a slow failure from a fast one.

The exit status encoding differs in a way that matters when the injector and the supervisor interpret results. A process that exits normally reports its own status; one killed by a signal is reported through the wait interface as terminated by that signal. Shells conventionally report 128 plus the signal number, so a kill by signal 9 appears as 137, but that arithmetic is the shell’s, not the kernel’s. The waitpid interface yields two distinguishable cases, exited or signalled, and a supervisor that only inspects the numeric status confuses a deliberate exit with a crash.

#include <signal.h>
#include <stdio.h>
#include <sys/wait.h>
#include <unistd.h>

int main(void) {
    pid_t pid = fork();
    if (pid == 0) {
        pause();                 /* wait forever for a signal */
        _exit(0);
    }
    kill(pid, SIGTERM);          /* default action: terminate */
    int status;
    waitpid(pid, &status, 0);
    if (WIFSIGNALED(status))
        printf("killed by signal %d\n", WTERMSIG(status));
    else if (WIFEXITED(status))
        printf("exited with %d\n", WEXITSTATUS(status));
    return 0;
}

What dies with the process and what does not

The kernel closes every file descriptor the process held, and the consequences are uneven. Advisory locks taken with flock are attached to the open file description and are released when the last descriptor referring to that description is closed, which happens at process death. POSIX record locks behave slightly differently: they are released when any descriptor for that file is closed by the process, and all of them go when the process exits, so a process that closes one descriptor loses locks it believed it still held. Both properties mean a kill can release a mutex the system was relying on for mutual exclusion — and the failure mode of released locks is silent concurrent access, not a visible error.

Network sockets are closed, and the kernel decides between an orderly close and an abrupt one. If unread data remains in the receive buffer when the socket is closed, the transport sends a reset rather than a FIN, per the behaviour specified for hosts in RFC 1122. The peer therefore sees an error rather than an end of stream, and clients that treat the two identically will not distinguish a killed server from a completed request stream. Sockets in the timed-wait state after an active close linger for a period of roughly a minute on common Linux configurations, and a deployment that kills and immediately restarts many processes can exhaust the ephemeral port range’s usable capacity even though the processes themselves are gone.

Buffered data in the page cache is not lost. Writes performed before the process died have been recorded by the kernel and are written to the block device regardless of the process’s fate, which is why kill faults do not corrupt filesystem metadata in the way a power fault can. Data still in the application’s own userspace buffer at the moment of SIGKILL is gone with no trace, and no amount of filesystem journaling recovers it.

Memory held by the process is reclaimed by the kernel, but the shared segments it attached to are not destroyed by the process’s death; a remaining attacher keeps them alive. Named shared memory persists until it is unlinked. The practical consequence is that a kill fault does not clean up resources that were not owned exclusively by the killed process, and a store whose replica state lives in a shared segment can appear to retain state across a restart that was intended to be cold.

Freezes, and the unreachable-node failure

Suspending a process with SIGSTOP produces the failure shape of a machine that has stopped responding: the process is scheduled out and never runs, peers see no response, but the transport layer keeps the connection open because nothing sent a FIN. Whether the peer notices depends entirely on whether it has a timeout at all, and what that timeout is set to. A client with no read deadline waits indefinitely. A client with a deadline derived from an expected latency will conclude the peer has failed after that period, retry, and possibly open more connections to a node that will never answer.

The container equivalent is the freezer controller in cgroup v2, which switches a whole subtree between frozen and not frozen by writing to the control group’s state; a frozen group, like a stopped process, is invisible as a failure to anything that only looks for connection closure or process exit. A monitoring check that tests process liveness by examining the process table will call a frozen process healthy, because the process still exists and is still marked running. The only useful liveness check for this failure mode is an application-level response within a deadline, which is why heartbeat protocols generally require an affirmative reply rather than the absence of an error.

Freezing is also the fault of choice when the desired shape is a network partition of one node without disturbing the rest of that node’s network stack, and it is the mechanism by which such a partition can be made from inside the node rather than by filtering packets elsewhere. The trade is that a freeze cannot be partial: the whole subtree stops. A multi-process service where only one component is frozen produces an asymmetrically degraded state that is hard to construct predictably in any other way, and which resembles the effect of a saturated thread pool more than it resembles a crash — a comparison made concrete by the resource-pressure faults described in resource exhaustion injection.

Reaping, orphans, and restart loops

When a parent dies before its children, the orphaned children are reparented to the nearest subreaper or to the init process of their PID namespace. The kernel keeps a terminated child in the zombie state until its parent collects the exit status, and a zombie consumes a process table entry but no other resource. A supervisor that never calls wait leaves zombies behind, and a container whose init process ignores its children accumulates zombies until the process limit for the namespace is reached, at which point new process creation fails and the container appears to break for reasons unrelated to its own code.

The observable outcome of a kill fault is therefore mostly determined by what the supervisor does next, not by the signal. A restart that reuses the same identity, reattaches to the same volumes, and re-registers the same address tests a different recovery path from one that gets a fresh identity and rebuilds its state. Both are worth testing, and they are separate experiments that belong to different parts of the same catalogue: the first is a continuous check that a restart returns a healthy replica, the second is a scenario that a game day can build on because it changes what the restarted process can assume about its own identity. The injector’s job is to make the chosen variant explicit — terminate, kill, or freeze; orderly or abrupt; same identity or new — because the innocent-looking word “restart” covers all of them.