Copy-Up, Whiteouts and Rename in Overlay Roots

A union mount presents several directories as one. Reads resolve to whichever layer has the name, highest priority first. Writes have nowhere to go in a lower layer, since layers are shared and immutable, so the filesystem must invent somewhere: the upper layer. The invention is called copy-up, and its details determine what a container can and cannot do to files it inherited.

Read is a lookup, write is a copy

When a name is opened for reading, overlayfs walks the layer stack from the top and stops at the first match. The inode returned is the underlying inode from that layer, with the overlay’s own dentry wrapped around it. Reads of file data go directly to the real file, so there is no duplication of content in the read path and no cost proportional to the layer stack beyond the lookup itself.

When the same name is opened for writing, and the resolved inode lives below the upper layer, overlayfs creates a copy in the upper directory: it allocates an inode there, copies the file’s data, and copies the metadata that matters (mode, owner, timestamps, extended attributes, ACLs). The open then proceeds against the copy. This happens on the first write, before any byte is modified, which is why the operation is described as copy-up rather than as a write-through.

The consequence for a large file is a latency spike at the first write regardless of how small the write is. Opening a 2 GB file for append copies 2 GB before the first byte of new data is stored, because the copy is what the write will land in. O_APPEND does not help: the file is copied, then the append happens in the copy.

There is a metadata-only fast path for operations that do not touch data. chmod, chown, and utimes on a lower file trigger a copy-up of metadata only: the data is not duplicated, but the upper layer gains an inode that shadows the lower one. Whether the implementation can avoid copying data in that case depends on the kernel version and on the filesystem backing the upper layer, and the observable behaviour is the same either way, since the file content is unchanged.

Directory copy-up is lazier. Creating a file inside a lower directory requires the directory to exist in the upper layer, so overlayfs creates the directory entry there without copying the entries beneath it. The merged readdir then combines the upper directory’s entries with the lower ones, subtracting any that are whiteouted.

Deletion without deleting

Removing a file that lives in a lower layer cannot remove it: the lower layer is shared by every container using that image, and it is immutable by contract. Instead, overlayfs creates a character device in the upper directory with the name of the file being removed, and with the whiteout device number (0, 0). The whiteout is a marker, not a deletion; during lookup, a whiteout in a higher layer means the name does not exist, and the search stops rather than falling through to the lower layer.

This representation is why ls in a container can show fewer files than the image contains without any of those files being gone, and why the upper directory of a container with a deleted 500 MB file grows by one inode rather than shrinking.

Opaque directories solve the adjacent problem. If a directory is removed and recreated, or renamed over, the upper directory must hide not only the specific names in the lower directory but all of them. An opaque marker on the upper directory means “do not merge lower entries here at all.” Without it, recreating /etc empty would still show the lower /etc contents, since a merge would find them.

The trusted.overlay.opaque extended attribute carries this marker. It requires CAP_SYS_ADMIN in the mount’s user namespace to set, which matters for unprivileged containers: an upper layer written by a process without that capability cannot express opacity through the standard attribute, and implementations fall back to other encodings or refuse the operation.

Rename is the hard case

rename(2) promises atomicity: a reader sees either the old name or the new one, never neither. Overlayfs has to preserve that promise across a stack of directories it does not control, and the cases split.

A rename entirely within the upper layer is handled by the upper filesystem and keeps its guarantees. A rename of a lower file to a new name requires the file to be copied up first, then renamed; the atomicity is now between the copy-up and the rename, and the source name’s disappearance is expressed as a whiteout at the old location. If the process crashes between the copy and the rename, the upper layer holds a duplicate of the file and a live lower entry, and the operation has not happened. The rename itself is still atomic at the moment it commits, but the work preceding it is not.

Renaming a directory that exists only in a lower layer is the case overlayfs historically refuses, returning EXDEV. The reason is that a directory rename would need to move every descendant, and descendants may live in several layers, some of them shared. Making the rename atomic would require copying the entire subtree up first, which breaks the “cheap rename” property that makes directory renames useful. Recent kernels handle some of these cases by redirecting the directory: an upper directory is created with a trusted.overlay.redirect attribute pointing at the lower path, so lookups inside it are forwarded. That makes the rename cheap and atomic but produces a filesystem where the same lower content is reachable under two paths, and where d_ino values and .. traversal differ from what a plain filesystem would give.

Applications detect the refusal rather than the mechanism. A rename returning EXDEV is the documented signal to fall back to copy-and-delete, which is what tools like mv do. The container-specific twist is that the fallback is not equivalent: copy-and-delete across a layer boundary loses hard links, changes inode numbers, and is not atomic.

What the merged view cannot preserve

Several POSIX behaviours depend on there being one filesystem with one set of inodes. Overlayfs satisfies them by approximation.

Hard links across layers. A file in a lower layer and another name for it in the same lower layer are hard links, and they stay linked while both are untouched. Once one of them is copied up, the copy is a new inode, and the link is broken silently. The upper file and the lower file then diverge on the next write, which is correct in the sense that each name keeps its own content, but wrong in the sense that a program which linked them expects them to stay one file.

Inode numbers. st_ino for a lower file is the overlay inode number, not the underlying one, and it is stable only while the overlay’s inode cache holds it. Tools that use inode numbers to detect hard links (find -samefile, rsync’s link detection, tar’s hard link encoding) can see two names for one file as distinct, or one file as two.

st_dev. All files in the overlay report the overlay’s device number, hiding the fact that different layers may sit on different filesystems with different block sizes and different available features.

Directory modification times. A write to a merged directory may update the upper directory’s timestamps but not the lower ones, so the merged directory’s mtime reflects whichever layer the implementation chose to report. readdir order is likewise not the order of any single layer.

O_DIRECT and other per-filesystem flags. A file opened with O_DIRECT is opened on the underlying file, so the alignment requirements are those of the lower filesystem. Copy-up of such a file involves reading through the page cache unless the implementation handles it specially, and the semantics of mixing buffered and direct access across the copy are not those of a single filesystem.

File locking and mmap sharing. Advisory locks are on the overlay inode, so two containers sharing a lower file but with separate upper layers do not contend, and two processes in one container do. MAP_SHARED writes to a lower file trigger copy-up, and the mapping is then of the copy; a mapping established before the copy-up still points at the lower inode, so the two views diverge.

Choosing the upper filesystem

The upper layer is an ordinary directory on an ordinary filesystem, and its properties leak through. The most consequential is whether that filesystem supports the extended attributes and device nodes the overlay needs: whiteouts require mknod, so an upper on a filesystem mounted nodev cannot represent deletions. The workdir must be on the same filesystem as the upper directory, since it is used for atomic staging during copy-up and rename, and a cross-device workdir is rejected at mount time.

xattr support is required for opaque and redirect markers, which rules out some network and FUSE filesystems as upper layers. Volatile mounts, where the upper is discarded at unmount, avoid the writeback cost but lose all container state. And the upper’s free space is the container’s writable budget, which is why quota is usually enforced on that directory rather than on the merged view; the memory cost of that writeback is charged separately, as described in Memory and CPU Limits Under the cgroup v2 Unified Hierarchy.

None of these are bugs in overlayfs so much as the cost of composing filesystems that were never designed to be composed. The mechanism is a lookup that falls through and a write that copies up; everything a program can observe follows from those two rules and from the fact that the lower layers are shared and must not change. A runtime that needs different semantics, such as block-level snapshots or per-container filesystem isolation, replaces overlayfs with a snapshotter that materialises a real filesystem per container, at the cost of storage and creation time. The assembly of the layer stack itself is described in From Image Manifest to Mounted Root Filesystem.