Syscall Filtering and the Process as a Security Boundary

An interpreter can refuse to execute an instruction. It cannot refuse to let the process call open. When the sandboxed code is native, or when a managed runtime is itself compromised, the only boundary left is the one the kernel enforces, and the kernel enforces exactly one thing about a process: whether a given system call is permitted.

The kernel as the last mediator

The process abstraction gives every program a private virtual address space and a table of open resources. Crossing from that space into the kernel happens only through the system call instruction, which on x86-64 is a transition through a designated entry point that switches to ring 0. A filter that runs at that entry point, before the syscall dispatcher examines the arguments, sees every transition. This is the narrowest mediation point available without hardware virtualization, and it is narrow in the sense that matters: there is no way to perform a privileged operation that avoids it.

Two mechanisms in Linux provide the filter. seccomp in strict mode allows only read, write, _exit, and sigreturn; it was designed for computational processes that need nothing else. seccomp in filter mode, added later and now the useful one, attaches a program written in classic Berkeley Packet Filter bytecode to the calling thread. The filter receives the syscall number and its six arguments as a struct seccomp_data and returns one of a small set of actions: allow, kill the process, kill the thread, return an error number to the caller, trap to a tracer, or notify a supervisor through a file descriptor.

Filters form a stack. Every filter attached to the thread is evaluated, in reverse order of installation, and the result is the action of highest precedence among them all, so a filter that returns allow cannot override another that returns an error. The stack cannot be loosened: a thread cannot remove a filter, and a filter installed by a parent before execve survives into the new program image. That combination is what makes the mechanism usable as a confinement boundary, because a supervisor can install a filter and then hand control to code it does not trust.

What the filter can actually see

The filter operates on the syscall number and six 64-bit argument registers. It cannot dereference pointers, because doing so would require the kernel to read user memory before deciding whether to read user memory, which is the wrong order for a security check.

The consequence is that argument filtering works only for arguments whose meaning is contained in the register value. The flags argument to openat is a bitmask in a register and can be tested: a filter can reject any openat whose flags include O_CREAT, denying file creation while permitting reads. The path argument is a pointer, and the filter can compare the pointer value against a range but cannot examine the string it points to.

A filter can therefore enforce “no execve”, “no socket in the AF_INET family”, “no ptrace”, “no ioctl on this descriptor number”. It cannot enforce “no opening of /etc/shadow”, because the path is not visible at filter time. Path-based policy requires either a supervisor that receives a notification and inspects the path from outside the process, or a different mechanism such as a mount namespace in which the path does not exist.

The argument-pointer limitation is the most common source of false confidence in seccomp designs. A filter that blocks open but not openat blocks nothing on a modern system; a filter that blocks write to descriptor 1 but not writev leaves the same channel open. The set of syscalls that reach a given resource is larger than it appears, and enumerating it correctly is the actual work.

Writing the filter by hand

The bytecode is a sequence of instructions over a 32-bit accumulator with a small scratch region. Loading the syscall number, comparing it, and returning an action is the whole idiom.

#include <linux/filter.h>
#include <linux/seccomp.h>
#include <stddef.h>
#include <sys/syscall.h>
#include <unistd.h>

struct sock_filter filter[] = {
    /* load the syscall number from struct seccomp_data */
    BPF_STMT(BPF_LD | BPF_W | BPF_ABS, offsetof(struct seccomp_data, nr)),
    /* allow write and exit_group */
    BPF_JUMP(BPF_JMP | BPF_JEQ | BPF_K, __NR_write, 0, 1),
    BPF_STMT(BPF_RET | BPF_K, SECCOMP_RET_ALLOW),
    BPF_JUMP(BPF_JMP | BPF_JEQ | BPF_K, __NR_exit_group, 0, 1),
    BPF_STMT(BPF_RET | BPF_K, SECCOMP_RET_ALLOW),
    /* everything else fails with EPERM */
    BPF_STMT(BPF_RET | BPF_K, SECCOMP_RET_ERRNO | EPERM),
};

The jump instructions take a true-target offset and a false-target offset counted in instructions, which makes hand-written filters error-prone: an off-by-one in a skip lands on the wrong return action, and the resulting filter usually allows more than intended rather than less. Production filters are generated by a library from a declarative list for this reason, and the generator is then part of the trusted base.

Returning an error is usually better than killing. A program that receives EPERM from an unexpected call may handle it, log it, or fall back; a program killed by SECCOMP_RET_KILL_PROCESS produces no diagnostic beyond a signal. The error-return action also keeps the filter’s presence from being a denial-of-service primitive against the supervisor.

Namespaces as the complement

Filtering decides which operations are permitted; namespaces decide what the operations mean. The two are complementary and neither replaces the other.

A mount namespace gives the process a private view of the filesystem, so a path that does not exist in that view cannot be opened no matter which syscall is used. A user namespace maps a range of user and group identifiers inside the namespace to a different range outside, so a process that appears to be uid 0 inside has no privilege outside. A network namespace gives a private set of interfaces and routing tables, so binding a port affects nothing shared. A PID namespace makes the process’s children invisible to the rest of the system and re-parents them to a designated init process when the namespace’s first process exits.

The combination is what containers are built from, and it is why a container’s isolation is mostly namespace-shaped rather than filter-shaped. A filter restricting the syscall set is still useful on top: it removes kernel attack surface by making entire families of calls unreachable, and it prevents a compromised process from using a call the namespace design did not anticipate. The reduction in attack surface is the honest justification, not a claim that the filter confines the process to particular resources.

Where confinement leaks

Several paths remain even with a good filter and full namespaces, and each has a distinct cause.

Descriptors are inherited. A process that receives an open descriptor to a directory can use openat relative to it to reach anything beneath, regardless of the path policy the supervisor intended. Descriptor passing over a Unix socket extends the reach further. A confinement design must therefore treat the descriptor table as part of the authority being handed over, which is the capability argument made in capability references applied at the kernel level.

Kernel interfaces that act as multiplexers are hard to filter usefully. ioctl takes a request code and a pointer, and the set of valid request codes is large and device-specific; a filter that allows ioctl at all allows nearly everything on the descriptors the process holds. prctl, fcntl, and the setsockopt family have the same shape. Allowing them selectively requires knowing the request-code ranges, and those ranges change between kernel versions.

Time and scheduling are shared. Even a perfectly confined process can observe wall-clock time and can consume CPU, so it can influence other processes on the same machine through contention. That is a covert channel rather than an escape, and it is treated in covert channels beneath the sandbox.

The filter is inherited across execve but not across a new process image started without it. A confined process that can create a child which the supervisor does not filter has escaped, which is why filters normally include a rule denying the syscalls that spawn processes without the supervisor’s involvement, or why the supervisor observes process creation through a different mechanism.

Choosing between the layers

The practical question is how much of the confinement to put in the filter, how much in namespaces, and how much above in the language runtime. The trade is between attack surface and expressiveness.

A tight filter with a small allowlist gives the smallest kernel attack surface and the least expressiveness; programs must be written to the allowlist, and the allowlist must be maintained as the program’s dependencies change. A permissive filter with namespaces gives expressiveness but leaves the full syscall surface reachable by a compromised process. A language-level sandbox gives the finest-grained policy, expressed over objects rather than descriptors, but it disappears entirely if the runtime is compromised.

The layered answer is that the language sandbox provides policy at the granularity of the application’s logic, the filter removes operations the application will never need, and the namespace defines what the remaining operations can name. Each layer’s failure mode is covered by the layer beneath, which is the only arrangement in which the sum is stronger than its parts.