Actor Mailboxes, Supervision, and the Limits of Location Transparency

The actor model replaces shared memory with message passing and replaces locks with a rule about execution: each actor processes one message at a time, from its own mailbox, and its state is reachable by nothing else. A language runtime may host millions of actors, schedule them on a small pool of threads, and move them between threads without the code noticing. The rule is what makes the movement safe — there is no point at which two threads touch the same actor state.

That rule removes one class of defect completely and leaves several others untouched, and the ones it leaves are the ones that decide whether an actor system behaves. Mailboxes grow. Messages arrive out of order across senders. A restart loses state. A remote send is transparent to write and opaque to failure.

Isolation is bought with a copy

Sending a message to an actor hands over a value, and in runtimes that guarantee isolation the value is copied into the receiver’s heap rather than shared. Erlang does exactly this for messages between processes, which is why a large term costs time proportional to its size to send, and why a sender cannot mutate what the receiver holds. The copy is the mechanism behind the guarantee that an actor’s state is private.

The cost shows up as a design constraint on message size. Passing a copied megabyte to a thousand actors copies a gigabyte, and none of the copies can be shared. The usual fix is to pass an immutable reference to data that lives outside the actor — a binary held in a shared read-only form — or to restructure so the data is read once by one owner rather than fanned out. Neither is free; both are visible in the message protocol rather than in the code that builds the payload.

On the receiving side, the mailbox is a queue with no built-in bound in most runtimes. A sender cannot be made to wait unless the protocol adds an acknowledgement and the receiver refuses to accept beyond a limit. This is the actor equivalent of an unbounded buffer, and it fails the same way: the process grows until the node runs out of memory, and the failure takes down everything scheduled on that node rather than only the actor that fell behind.

Backpressure therefore has to be a protocol decision. The workable shapes are a bounded number of outstanding requests tracked by the sender, a pull model in which the consumer asks for the next item, or a mailbox limit that causes the sender to fail fast instead of buffering. A system that relies on the runtime to slow senders is relying on a mechanism the runtime does not provide.

Supervision is a tree, not a retry loop

Failure handling in the model is structural. An actor that fails is restarted or terminated by its supervisor, which is itself an actor, and supervisors form a tree rooted at the application. The failure is contained at the level that knows how to recover rather than handled inline at the point of the error.

Erlang’s OTP supervision behaviours define the common strategies and they are worth naming because they describe distinct recovery semantics. Under one_for_one, a failing child is restarted alone. Under one_for_all, all siblings restart, which is correct when they share an invariant that one failure invalidates. Under rest_for_one, the failing child and the siblings started after it restart, which encodes a dependency order: later children depend on earlier ones. The supervisor is configured with a maximum restart intensity — a count within a time window — and exceeding it terminates the supervisor itself, escalating to its parent. That escalation is the part that prevents a persistent fault from producing an endless restart loop that looks like activity and delivers nothing.

Restart means re-running the child’s initialisation, so state held in the process is lost unless it was written somewhere durable or handed to the restarted process by the supervisor. A supervisor can pass the arguments it was started with into every restart, which recovers a fixed starting state but not accumulated progress: the supervisor’s own state is the child list, and it has no place to keep values that change. State that has to survive a restart therefore lives outside the process — in a store, or in another process that the restarted child asks for it. A system that assumes a restart preserves state has a recovery path that silently resets every counter and every in-flight operation.

%% the child is restarted with the arguments the supervisor was started with
init(State) ->
    {ok, {{one_for_one, 5, 10},
          [{worker, {worker, start_link, [State]},
            permanent, 5000, worker, [worker]}]}}.

The intensity tuple means five restarts within ten seconds are tolerated; the sixth terminates the supervisor. The numbers should be chosen against the time a transient fault takes to clear, not copied.

Failure signalling matters as much as restarting. A link propagates an exit to a linked actor, so a crash reaches its supervisor; a monitored actor instead delivers a message on exit and leaves the monitor running. Exit signals can be trapped and turned into ordinary messages, which is how a graceful shutdown drains a mailbox before terminating. The distinctions are small and each one is a different shutdown contract.

At-most-once delivery, and what request-reply needs

Message sending is asynchronous and, in its local form, reliably delivered if the recipient is alive; the guarantee is not end-to-end acknowledgement. Erlang’s documented guarantee is ordering between a specific pair of processes: messages sent from one process to another arrive in the order they were sent. Across different senders there is no order, and a remote send may fail without notifying the sender.

This is at-most-once from the sender’s perspective, and a request-reply protocol has to be built on top of it. A correlation identifier is generated per request, the reply carries it, and the requester holds a timer. When the timer fires, the requester does not know whether the request was lost, the reply was lost, or the recipient is still working, so the default outcome is a retry and the recipient must be prepared to see the same request twice. The identifier is what allows the recipient to recognise the duplicate, and the requester’s timeout is what bounds the wait.

A blocking request inside an actor is the second problem. If the actor waits for a reply by receiving, it stops processing the rest of its mailbox, and any message that depends on that actor’s progress — including the reply it is waiting for, if the path loops back — deadlocks. The model’s answer is to keep the reply asynchronous: the actor records the continuation in its own state, returns to the mailbox, and completes the work when the reply arrives. That turns every request-reply into a small state machine, which is the real cost of using actors for orchestration.

Selective receive and the cost hidden in the queue

A receiving actor can match on message shape rather than taking the head of the queue. Erlang’s receive scans the mailbox from the front, takes the first message that matches the pattern, and leaves unmatched messages in place. For a mailbox with m messages and a pattern that matches the m-th, the scan costs m comparisons, and unmatched messages stay to be scanned again by the next receive.

The arrangement is convenient for protocol code: a reply with a known correlation identifier can be received directly, ignoring unrelated messages. It is also a performance cliff. An actor that holds a large backlog and performs a selective receive on every iteration does work proportional to the backlog per message, which turns a queue into a quadratic cost. The mitigation is to keep mailbox backlogs small, to receive in the order messages arrive with a catch-all clause that dispatches internally, or to separate concerns into more actors so that each mailbox stays short.

Location transparency and its boundaries

An actor address looks the same whether the actor runs on the local node or on another node, and the send syntax does not change. That uniformity is the property the model advertises, and it is real at the level of code.

It is not real at the level of failure. A local send either delivers or the recipient no longer exists, and the failure is immediate and observable. A remote send involves a connection that can break at any point around the send, so the sender cannot distinguish “delivered and the node died” from “never delivered”. Latency changes by orders of magnitude, and a design that assumed local sends are cheap becomes a design whose mailbox fills at the rate of the network. The serialization boundary also appears: a closure or a reference to local state cannot cross the node boundary, so the message protocol must consist of serializable values even when every actor in development runs locally.

Systems that treat the boundary as invisible end up with actors that hold references to remote state and a failure mode where a partition looks like a slow actor. Systems that treat it as a visible assumption — a supervisor per node, a bounded outstanding-request count for remote calls, explicit handling of node-down events — keep the ergonomics and pay only for the failure model.

Actors and event-driven integration share the message-passing shape but not the guarantees: an actor address is a point-to-point channel with per-sender ordering, while a broker is a durable log with its own delivery semantics. The space-based style reaches for similar isolation by replicating data rather than by serialising access through a mailbox.