Assembling a Container Root with pivot_root and Overlay Mounts

A container’s filesystem has three requirements that pull against each other: the root must be the namespace’s root and not merely the process’s, the image must be shared between many containers without being copied, and each container must be able to write. chroot satisfies none of the three properly — it changes the process root inside a namespace, it requires a copy per container, and it leaves the host tree reachable. The pair of mechanisms that do satisfy them is pivot_root, which moves the mount namespace’s root, and overlayfs, which stacks a writable layer over a shared read-only one.

Why the root must move in the namespace

pivot_root(new_root, put_old) requires that both arguments are directories, that new_root is a mount point, that put_old is underneath new_root, and that new_root is not on the same mount as the caller’s current root. It then detaches the old root mount, makes new_root the root of the namespace, and stacks the old root at put_old so that it can be unmounted afterwards. The operation is atomic with respect to other processes in the namespace: no path walk observes an intermediate state where neither mount is the root.

The requirement that new_root be a mount point is the reason a container runtime bind-mounts the assembled tree onto itself before pivoting. A bind mount of a directory creates a mount whose root is that directory, which gives pivot_root the object it needs and simultaneously severs the tree from the parent’s mount structure — the new mount is a separate mount, so propagation from the host stops at it if the tree was made private first.

/* new_root is a bind mount of the assembled image tree. */
mkdir("/run/c/oldroot", 0755);
if (syscall(SYS_pivot_root, "/run/c", "/run/c/oldroot") < 0)
        die("pivot_root");

/* The old root is now mounted at /oldroot; drop it. */
umount2("/oldroot", MNT_DETACH);
rmdir("/oldroot");

The chdir("/") that appears in most implementations of this sequence is defensive rather than strictly required: pivot_root repoints the root and current working directory of every process in the namespace that pointed at the old root, so a caller sitting in a directory that has just moved to /oldroot is relocated without having to move itself. The MNT_DETACH flag is what makes the unmount reliable. An ordinary umount fails with EBUSY if any process still holds a reference — an open file descriptor into the old tree, a memory mapping of a file there, or a working directory in a process that was not in the namespace at the moment of the pivot. The lazy form removes the mount from the namespace immediately and defers teardown until the last reference drops, so the container proceeds while the mount lingers, invisible but alive.

The failure conditions are specific. pivot_root returns EINVAL if new_root is not a mount point or if put_old is not underneath new_root; ENOTDIR if either argument names something other than a directory; and EPERM without CAP_SYS_ADMIN in the user namespace that owns the mount namespace. The pairing with unshare(CLONE_NEWNS) follows from those conditions: a process that has not created a mount namespace has the host’s root as its current root, and pivoting there would move the host’s root rather than a copy.

The overlay as a stack of directories

Overlayfs presents a union of directories. The lower layers are read-only and may be shared; the upper layer is writable and private to the container; the merged view is what gets mounted. Lookups walk the layers from the top down and stop at the first match, with one important exception: a directory present in several layers is merged, so its contents come from all layers, while a file present in several layers is taken from the highest one.

Writes go to the upper layer. Opening a file for writing that exists only in a lower layer triggers a copy-up: the file is copied into the upper layer, preserving ownership, mode, and extended attributes, and the write proceeds against the copy. The copy is whole-file, not per-block — overlayfs has no knowledge of the lower file’s block layout — so the first write to a large file costs a full copy of that file. Subsequent writes hit the upper copy. Deletion of a lower file creates a whiteout in the upper layer: a character device with zero device numbers, which the merged view interprets as “this name does not exist” and which hides the lower entry.

mount -t overlay overlay \
  -o lowerdir=/var/lib/images/base,upperdir=/run/c/upper,workdir=/run/c/work \
  /run/c/merged

The workdir is not optional. Overlayfs needs a directory on the same filesystem as upperdir for temporary files used during copy-up and rename; the requirement exists because rename of a merged directory may need to copy the directory’s contents up first, and doing that without a staging area would expose partially copied state. The kernel rejects a mount whose workdir is on a different filesystem from upperdir with EINVAL.

Multiple lower layers are supported and are separated by colons, with the leftmost being highest in the stack. That is how layered images are represented: each image layer is a directory, the layers are stacked in reverse order of application, and a container’s upper directory holds only what it changed. Each additional layer adds a directory to every lookup that reaches it, so the cost of resolving a path is proportional to the number of layers the name is absent from — a lookup for a file in the bottom layer of a ten-layer stack reads ten directories.

The semantics overlayfs does not preserve

Hard links across layers break. A file in a lower layer that is hard-linked to another name in the same lower layer keeps that link, but after copy-up the upper copy is a new inode with a single link, so the two names diverge. Creating a hard link to a lower-layer file does not fail; it triggers a copy-up first, and the new link is created against the upper copy, so the operation succeeds but produces a link to a different inode than the original lower name. An application that relies on link counts for atomic replacement observes the divergence rather than an error.

st_dev and inode numbers are not stable across the merged view. A directory in the merged view reports the overlay’s own device number, while a non-directory reports the device number of the underlying upper or lower filesystem that actually holds it, and its inode number is the underlying filesystem’s. The result is that the pair (st_dev, st_ino) for a file can change when the file is copied up, which breaks anything that persists an inode identity, such as a backup tool comparing inode numbers or a database that assumes inode stability. The instability is inherent: two containers sharing the same lower layer see the same underlying inode, and an overlay cannot both report that inode and remain independent of the layer.

mmap interacts with copy-up in a way that is easy to miss. A shared mapping that permits writes requires the descriptor to have been opened for writing, and opening a lower-layer file for writing is what triggers the copy-up, so such a mapping is established against the upper copy. The copy therefore happens at open time, not lazily at the first fault. A mapping created read-only from a read-only descriptor stays read-only, and a write through it faults rather than reaching the lower file.

Assembling the rest of the tree

A root filesystem is not enough. The container needs /proc mounted in its own PID namespace, /dev with a private devpts instance, /sys either read-only from the host or a fresh sysfs mount in the container’s network namespace, and /dev/shm as a private tmpfs. Each mount is performed after the namespace creation and before the pivot, because after the pivot the paths that the mounts would be attached to no longer exist in the host tree.

/dev is the one that repays attention. A tmpfs mounted there with nosuid,mode=755 starts empty, and the runtime must populate it with the standard nodes — null, zero, full, random, urandom, tty, console — using mknod or bind mounts from the host. A new devpts instance mounted at /dev/pts with newinstance gives the container its own pty numbering, so a terminal opened inside the container does not collide with one on the host. Bind-mounting the host’s /dev instead of building a private one hands the container every device node on the machine, which is a complete escape if the container has the capability to open them.

The ordering of mount operations matters for the same reason propagation matters. Every mount performed after the tree is made private stays local. Every mount performed before is inherited by every namespace created from the parent, and if the parent is shared, propagated outward. The sequence in a runtime is therefore: create the mount namespace, make the tree recursively private, build the image tree and its bind mounts, pivot, detach the old root, then drop capabilities. The mount namespace mechanism itself is described in Mount Namespaces and the Propagation of Mount Events, and the capability drop in Capabilities and the Decomposition of Root Privilege; the pivot is the step that turns the accumulated mounts into a coherent root.

Failure modes at teardown

An overlay mount holds references to its lower and upper directories, and the mount cannot be torn down while any file in the merged view is open. A container that leaves a process with a descriptor into the merged tree keeps the overlay alive after the namespace is gone, which in turn keeps the upper directory’s files from being deleted. The lazy unmount path handles the namespace side; the filesystem side requires the descriptor to close, and a runtime that deletes the upper directory before unmounting finds the deletion deferred rather than immediate.

The old root’s detach is similarly deferred. If the runtime process itself has a working directory or an open descriptor in the old root (a log file opened before the pivot, for instance), the mount stays alive with its full tree, and the host filesystem that the container was supposed to lose remains reachable from the container’s own descriptor table. The container can still reach it through openat on that descriptor, exactly as described in Chroot Confines Path Resolution, Not the Process. Closing every descriptor that refers to the old tree before the pivot is the only reliable way to avoid it, and it is a step that is easy to omit because nothing fails when it is.