Supervision Trees and the Shape of a Failure Domain

An actor that crashes takes its state with it and leaves a mailbox full of messages addressed to a process that no longer exists. Something must decide what happens next: restart the actor with fresh state, discard it, or escalate. The mechanism that makes this decision is a supervisor, and the arrangement of supervisors is a tree whose branches are failure domains.

The design premise is that a process’s state is reconstructible. If it is, the correct response to a crash is to restart from a known good state rather than to attempt repair of a corrupted one. The premise is strong, and where it fails, as with a process holding state that exists nowhere else, supervision cannot help, and the state has to be externalised before the crash rather than recovered after it.

Restart strategies and what they preserve

A supervisor starts children according to a strategy and restarts them according to a policy. The strategy determines startup order and dependency: one-for-one starts children independently; one-for-all restarts every child when one fails; rest-for-one restarts the failed child and the children started after it, preserving the assumption that later children depend on earlier ones.

The distinction that matters is how much state is discarded. One-for-all is appropriate when the children share an invariant that cannot be restored piecewise, as with a pair of processes that must agree on a generation number. It is expensive, because a failure in a leaf takes down siblings that were healthy. One-for-one is the default for independent children and keeps the blast radius to a single process. Rest-for-one encodes an ordering assumption: children started later depend on children started earlier, so a failure in an earlier child invalidates the later ones but not the reverse.

None of these strategies preserves state. A restart is a fresh start with the child’s initial arguments, so any state accumulated at runtime is gone. A supervisor that restarts a process holding a cache will produce a process with an empty cache, and the resulting load spike is a consequence of the restart policy, not of the crash.

Restart intensity is a rate limit on the supervisor

A supervisor does not restart indefinitely. It carries a maximum restart count and a time window; if the count is exceeded within the window, the supervisor itself terminates. In Erlang’s OTP the parameters are conventionally named for the maximum number of restarts and the period in seconds, and the supervisor gives up when the limit is crossed. A restarted child counts as a restart only if it terminates: a child that is started again on explicit request, without its predecessor having died, does not consume the budget.

The mechanism is a sliding count, not a lifetime total. A supervisor permitted a small number of restarts per minute can restart the same child repeatedly for a year as long as the failures stay sparse. The count exists to distinguish a transient fault, such as a message that arrived malformed once, from a persistent one, such as a dependency that is down or a code path that fails on every input. Restarting a child that deterministically crashes on startup in a tight loop burns CPU and floods logs; the intensity limit converts that into a supervisor failure, which propagates upward.

The choice of numbers is a policy about how long an outage should be tolerated before escalation. A generous limit keeps a flaky dependency from taking down the tree, at the cost of delaying the escalation that would have failed over to a working node. A tight limit escalates quickly, at the cost of treating a genuinely transient fault as fatal.

Escalation and the recursion

When a supervisor exceeds its intensity, it terminates, and its own supervisor sees a child failure and applies its own policy. The recursion is the point: the failure climbs until it reaches a supervisor whose policy can handle it, or until it reaches the root, where the application terminates.

This gives a natural encoding of layered recovery. A leaf supervisor restarts individual workers. Its parent, seeing repeated leaf failures, restarts the entire subsystem. The root, seeing repeated subsystem failures, restarts the application. Each level has a coarser remedy and a longer time constant, and the tree’s shape is the statement of which remedy applies at which granularity.

The failure to avoid is a supervisor that restarts a child which will always fail. Since the supervisor cannot inspect the child’s state, it cannot tell a poisoned input from a transient error, and the intensity limit is the only defence. A supervisor with an unlimited restart policy and a deterministically failing child is a busy loop that also destroys every other child in a one-for-all group.

Failure domains and the cost of the boundary

A failure domain is the set of processes that a single crash can take down. In a one-for-one tree it is one process plus its mailbox. In a one-for-all group it is the whole group. Across a distribution boundary it is one node.

The boundary is drawn by choosing where to put a supervisor, and every boundary has a cost. A boundary that is too fine produces many small supervisors, each with its own restart accounting, and the recovery is local but the failures never escalate to a remedy that could have fixed the underlying cause. A boundary that is too coarse means a single bad process restarts a large subsystem, and the resulting recovery time is proportional to the subsystem’s startup cost.

The usual guidance is to place a supervisor at each point where a genuine dependency relationship exists, and to keep siblings independent. Two processes that share nothing but a parent should be one-for-one siblings; two processes where one cannot function without the other’s output should be ordered by rest-for-one or combined into a single process.

What a crash destroys beyond the process

The process’s mailbox is lost. Messages already sent to the dead address are not redelivered, because the send is asynchronous and at-most-once, a point covered in Mailbox Semantics and Ordering Guarantees in the Actor Model. A request that was in flight when the crash happened has no reply and no notification; the requester waits until its own timeout, if it has one. Supervision restores the service, not the individual request.

Monitors and links are also affected. A process that was monitoring the dead one receives a down message, which is how a supervisor learns of the failure in the first place. Any process that linked to the dead one and did not trap exits dies with it, which is the mechanism by which a failure propagates along a link rather than up a tree. A link is a bidirectional failure relationship; a monitor is a one-way notification. Getting the direction wrong turns a supervised failure into a cascading one.

Resources held outside the runtime are not released by a crash in the way a destructor would release them. A file descriptor is closed when the owning OS process exits, but a database connection held on a remote server is not: the peer, if it notices at all, notices only as a half-open connection, and it lingers until its own timeout expires or a keepalive probe fails. A restart that immediately reconnects to the same server adds a connection to the set that is already leaking, and a crash loop can exhaust the server’s connection limit, so the failure domain extends beyond the tree.

The supervision boundary and the rest of the model

Supervision is the actor model’s answer to the question every concurrent system faces: namely what happens when a unit of concurrency fails. Threads have no equivalent, because a thread’s failure is either caught by a handler inside it or terminates the process; there is no parent with a policy. Structured concurrency provides a partial analogue, in that a task scope propagates a child’s failure to the parent and cancels siblings, which is a one-level escalation with no restart policy; see Structured Concurrency and the Task Scope.

The difference is what the failure is allowed to mean. In a task scope, a failure is an exception that propagates and the scope’s remaining tasks are cancelled; the computation is abandoned. In a supervision tree, a failure is an event that the supervisor handles, and the computation continues with a fresh process. The first is appropriate when the whole operation is meaningless without its parts; the second is appropriate when the service must continue despite losing a part.

That distinction sets the limit of supervision. It cannot make a stateful process stateless, cannot make a non-idempotent operation safe to retry, and cannot recover state that was never persisted. It provides a policy for restarting, a rate limit on that policy, and a structure for deciding which parts of the system fail together. Everything else is the application’s responsibility, and a tree with a well-chosen shape makes that responsibility visible in the source rather than in the aftermath of a failure.