Supervising Syscalls with Seccomp User Notification

SECCOMP_RET_USER_NOTIF is the one action that lets a filter return a decision it cannot compute. Instead of allowing or denying, it stops the calling thread inside the syscall and delivers a message to a supervisor, which answers later. The mechanism turns a fixed 64-byte input into an arbitrary computation, at the cost of two context switches per intercepted call and a dependency on a second process staying alive.

Attaching a listener

A filter that returns SECCOMP_RET_USER_NOTIF is installed with SECCOMP_FILTER_FLAG_NEW_LISTENER. The seccomp call then returns a file descriptor for the listening end of the notification channel rather than zero:

int listener = seccomp(SECCOMP_SET_MODE_FILTER, SECCOMP_FILTER_FLAG_NEW_LISTENER, &prog);

That descriptor is passed to the supervising process, usually by SCM_RIGHTS over a Unix socket, before the sandboxed process starts. Only one listening filter can be installed for a thread, and if its descriptor is closed, later calls routed to notification fail with ENOSYS rather than blocking. Notification is an action in the same ranking as the other verdicts, so a supervisor only sees the calls that no stricter filter in the chain already refused, as described in Action Precedence When Seccomp Filters Stack. The kernel deliberately does not block forever on a missing supervisor: a dead supervisor must not wedge the sandbox, so the failure is immediate and takes the form of a call that appears unimplemented.

The notification record

The supervisor blocks in ioctl(listener, SECCOMP_IOCTL_NOTIF_RECV, &req), where req is a struct seccomp_notif. Its fields are the same information the filter saw plus an identifier:

• id is a 64-bit cookie unique to this notification. It is the handle used to answer and to validate, and it is not a process or thread identifier.
• pid is the thread ID of the calling thread in the supervisor’s PID namespace.
• flags is reserved and zero.
• data is a struct seccomp_data: the syscall number, architecture, instruction pointer, and six argument words, exactly as the filter received them.

The supervisor answers with ioctl(listener, SECCOMP_IOCTL_NOTIF_SEND, &resp), where struct seccomp_notif_resp carries the same id, a val for a successful return value, an error for a negative errno, and flags. Setting SECCOMP_USER_NOTIF_FLAG_CONTINUE tells the kernel to execute the syscall as the caller requested; without it, the kernel skips the syscall and returns val as though it had succeeded.

SECCOMP_IOCTL_NOTIF_ID_VALID checks whether a notification is still pending. The check is necessary because the calling thread can be interrupted, by a signal or by exit_group, between the supervisor’s RECV and its SEND, and answering a notification whose thread has gone produces ENOENT and a decision applied to nothing. Supervisors that read the caller’s memory first must validate the id after the read, not before, since the read is the slow step and the window it opens is exactly the window the check exists to close.

Why continuing is usually the wrong answer

The obvious supervisor design is: receive the notification, read the path from data.args[1] with process_vm_readv, decide, and either deny with error or allow with CONTINUE. That design contains the race from Why a Seccomp Filter Cannot Inspect Syscall Arguments in Memory in full. Between the read and the CONTINUE, the calling thread’s other threads can rewrite the buffer at that address, so the syscall the kernel finally executes may name a different path than the one the supervisor approved. The id validation does not help: the thread is still alive and the notification still pending, only the memory changed.

The sound pattern is to never let the caller’s own syscall perform the operation. For openat, the supervisor opens the file itself, subject to its own privileges, and injects the resulting descriptor into the caller with SECCOMP_IOCTL_NOTIF_ADDFD. The caller receives a file descriptor number for a file the supervisor chose, and the syscall is answered with that number rather than continued. There is no second resolution of a path, so there is no second chance to change it.

SECCOMP_IOCTL_NOTIF_ADDFD takes a struct seccomp_notif_addfd with the notification id, a srcfd in the supervisor, a newfd in the caller (or SECCOMP_ADDFD_FLAG_SETFD to request a specific number), and SECCOMP_ADDFD_FLAG_SEND to install the descriptor and answer the notification in one operation. Installing and answering atomically is the point: a separate ADDFD followed by a SEND leaves a window in which the descriptor exists but the caller has not been told its number, and a multithreaded caller can close or reuse it.

The supervisor can also answer with a fabricated success value for syscalls whose return value it can compute without doing the work, such as a getpid that returns the supervisor’s notion of the caller’s identity, but fabricating a descriptor number for openat is not possible, because the number must refer to a real open file.

Direction of authority

A supervisor runs outside the sandbox and is not subject to its own filter. That asymmetry is what makes the design work and is also its main hazard. The supervisor’s policy is ordinary code with ordinary privileges; if it decides to allow an operation because the path matches a prefix, it is performing a string comparison on a path it resolved itself, and the soundness of that comparison depends on it having resolved the path through the same namespace the caller sees, with the same working directory, and with symlinks handled the way the caller would.

The practical way to make this tractable is to give the supervisor a file descriptor for the sandbox’s root and resolve every path relative to it with openat2 and RESOLVE_IN_ROOT or RESOLVE_BENEATH. Then the supervisor’s notion of the filesystem is the sandbox’s, and the prefix comparison happens on a resolved path rather than on a string.

A supervisor that answers CONTINUE for anything but trivially safe calls is not a sandbox. It is an auditing layer that reports what the sandboxed code attempted, and it should be described that way.

Cost and capacity

Each intercepted call costs the calling thread a sleep, the supervisor a wakeup, and the reverse on the answer. On the order of microseconds per call at best, and considerably more when the supervisor must perform a syscall of its own to decide. That is one to two orders of magnitude more than evaluating a filter program of a few hundred instructions, which is why notification is applied to a narrow set of calls such as openat, mount, and connect to specific ports, rather than to the syscall set as a whole. A filter that routes every syscall to a supervisor converts a program into a client of a single-threaded server, and the server’s throughput becomes the program’s.

Concurrency adds a design constraint. Notifications from different threads of the sandbox arrive on the same listener descriptor and can be received in any order. A supervisor that must serialize, because its decisions consult shared state, receives them one at a time and becomes a bottleneck; a supervisor that handles them concurrently must be careful that the sandbox cannot use ordering to confuse it. The kernel does not promise that notifications arrive in the order the syscalls were issued across threads.

The SECCOMP_IOCTL_NOTIF_RECV call takes a zeroed struct seccomp_notif whose size the supervisor learns from seccomp(SECCOMP_GET_NOTIF_SIZES, 0, &sizes), and it blocks by default. A supervisor that must also service other descriptors (a control socket, a timer) either puts the listener in non-blocking mode and polls, or dedicates a thread to it. Both are ordinary Unix programming, and both add latency compared to a filter verdict returned inside the kernel.

Failure modes

If the supervisor dies, pending notifications fail with ENOSYS and the sandboxed process sees a syscall that appears unimplemented. For openat that is a clear failure; for a call the program treats as optional, it may silently degrade into a fallback path the supervisor never intended to allow.

If the sandboxed thread is killed while a notification is pending, the supervisor’s SEND fails with ENOENT and the descriptor it may have opened for ADDFD leaks unless it is closed. Supervisors that open a file per notification must close on every path, including the error paths.

If the supervisor itself is inside the sandbox’s thread group, its own syscalls are intercepted and it deadlocks waiting for itself to answer. The supervisor must be a separate process, and the listener descriptor must reach it before the filter is installed on the threads it will supervise.

The documented position on the mechanism is narrower than its capabilities suggest, and the manual page states it directly: user notification must not be used to make security policy decisions, because CONTINUE is inherently race-prone. Its intended use is a more privileged supervisor performing operations on behalf of a less privileged target, where the supervisor is confident that another mechanism (the kernel’s own permission checks, an LSM, or a mount namespace) will refuse the operation if the arguments are rewritten to something unsafe before the kernel executes it. That is a statement about where the trust boundary sits, and it explains why the descriptor-injection path is the part of the interface worth building on.

A second structural limit is that a filter returning a higher-precedence action overrides the notifier. If the sandboxed process can install its own filter that returns KILL_PROCESS or ERRNO, it can bypass a supervisor that was relying on USER_NOTIF to see a call, so the sandbox must also deny seccomp and prctl, and deny them with EPERM rather than ENOSYS.