Reading a Directory as a Stream: readdir and Its Weak Guarantees

Listing a directory feels like reading a fixed collection: open it, iterate, close. The interface is a cursor over a file that other processes may be modifying at the same time, and the standard says almost nothing about what a listing contains when that happens. Two listings of the same directory, taken moments apart while an unrelated process creates and deletes entries, can differ in ways that no amount of care at the reading side can prevent.

The call beneath the call

readdir is a library function over getdents64, which reads raw records from the directory file into a caller-supplied buffer. The kernel returns a sequence of variable-length records with the same layout as the on-disk entries, each carrying an inode number, a record length, a name length, a type, a name, and an offset.

The offset field is the part that matters. Each record carries the position of the next record within the directory, an opaque cookie whose only meaning is that passing it to lseek on the directory descriptor positions the stream at the record after the one that produced it. telldir returns the cookie for the current position and seekdir restores it. The cookie is not a byte offset in general: ext4’s hash-indexed directories use a value derived from the hash and the block, and some filesystems encode a version counter alongside the position.

A program that stores cookies and seeks back later is assuming the directory has not changed in a way that invalidates them. The standard makes no such promise. On a directory that has been compacted, a stored cookie may point past the end of the directory or into the middle of a record, and the resulting behaviour is undefined. Cookies are valid for the duration of one open stream and are not a durable identifier.

readdir returns a pointer into a buffer owned by the stream, valid until the next call on the same stream. Copying the name into program-owned storage is therefore mandatory if it is to outlive the iteration, and it is the reason a naive implementation that collects pointers and sorts them afterwards produces a list of identical or corrupted strings.

What the type field is worth

The record’s type field, exposed as d_type in the dirent structure, lets a lister skip a stat per entry. Its values are DT_REG, DT_DIR, DT_LNK, DT_FIFO, DT_SOCK, DT_CHR, and DT_BLK, plus DT_UNKNOWN.

DT_UNKNOWN is not an error. It is a legitimate answer from filesystems that do not store the type in the entry, because some on-disk formats never did and some network protocols do not transmit it, and a lister that needs the type must fall back to stat when it sees it. Code that compares d_type against DT_REG without handling DT_UNKNOWN silently skips every regular file on such a filesystem.

The type is a hint and can disagree with the inode in a damaged directory. The authoritative answer comes from the inode’s mode word, described in What an Inode Stores, and Why It Stores No Name, and the hint exists only to avoid the second read.

Iteration during modification

The standard’s position is narrow: whether a subsequent readdir returns an entry for a file that was added or removed after the stream was opened is unspecified. It says nothing about ordering, nothing about duplicates, and nothing that amounts to a snapshot. A name removed and recreated during the scan may appear twice or not at all, and a rename within the directory can move a record from a position the cursor has not reached to one it has.

The mechanism behind that weakness is that the cursor is a position in a file whose records are edited in place. A new entry is normally appended at the end of the directory’s data, so a scan that has already passed that point never sees it. A deleted entry is merged into its predecessor by extending the predecessor’s record length, which is invisible to a cursor sitting exactly on that predecessor but not to one whose position was computed before the merge. Filesystems mitigate the problem by never shrinking a directory, which keeps the file’s length stable, but no in-place scheme makes record merging invisible to a concurrent cursor.

#include <dirent.h>
#include <fcntl.h>
#include <string.h>

static int drain(const char *path)
{
    DIR *d = opendir(path);
    struct dirent *e;
    if (!d)
        return -1;
    while ((e = readdir(d)) != NULL) {
        /* Only the two reserved names; a leading dot alone would skip real files. */
        if (strcmp(e->d_name, ".") == 0 || strcmp(e->d_name, "..") == 0)
            continue;
        unlinkat(dirfd(d), e->d_name, 0);      /* removal during iteration */
    }
    closedir(d);
    return 0;
}

That loop is a common way to empty a directory, and it is correct enough in practice because the removal of the current record does not move the cursor backwards. It is not guaranteed by the standard, and a filesystem that compacts on delete would make it skip entries. The safe version collects the names first, closes the stream, and removes them afterwards, at the cost of memory proportional to the entry count.

Why sorting requires a full read

ls sorts its output, and it does so by reading every name before printing any of them. There is no interface for reading a directory in sorted order, and none of the common Unix filesystems keeps a directory sorted by name for iteration: entries sit in creation order, or in hash order once an index exists, and the order is an implementation detail that changes when an entry is deleted and its slot reused.

The consequence is memory. Listing a directory of a million files requires holding a million names, on the order of tens of megabytes, before the first line of output. Tools that avoid this either print unsorted, which is what ls -U does, or accept the memory.

The default order also has a locality property that makes it worth keeping. Entries created together have adjacent inode numbers on a traditional allocator, so iterating in directory order and then stating each name produces mostly sequential metadata reads. Sorting by name before stat destroys that locality, so a walk that sums sizes in directory order issues its metadata reads in an order the device handles better than one that sorts first.

The two names that are always present

. and .. are returned by readdir and are filtered by every consumer. Their presence follows from the directory format: they are ordinary entries, and the reader does not special-case them, so the lister must.

.. in a listing refers to the parent by inode number, and a program that resolves it by opening the name re-resolves the parent through the tree rather than through the recorded number. On a directory that has been renamed since the stream was opened, the two can disagree: the entry’s inode number is the parent at the time of the listing, and the name resolves to whatever is at that path now. This is one more instance of the general rule that a directory listing describes the past.

Streams, descriptors, and the shared offset

opendir opens a descriptor on the directory and wraps it in a DIR structure holding a buffer. Because it is an ordinary descriptor, it participates in all the semantics described in Open File Descriptions and the Offsets They Share: the position belongs to the description, so a duplicated descriptor shares it, and fdopendir on a descriptor the program already held takes over that position rather than starting at the beginning.

A program that calls readdir on a stream and getdents64 directly on the same underlying descriptor interleaves two readers over one offset and gets a listing split between them. Mixing the interfaces requires a rewinddir or a seekdir to a known cookie between the two.

Closing the stream closes the descriptor. closedir is required even when the iteration ended with an error, and the memory the buffer occupied is released only then. A process that opens a stream per directory in a deep recursion holds one descriptor per level, which is why recursive walkers that keep streams open across the recursion can exhaust the descriptor limit on a tree deeper than the limit; closing the stream before descending and reopening afterwards avoids it, at the cost of re-reading the directory’s blocks.

When the listing is a snapshot

A listing that must be stable while the directory changes cannot be built from readdir alone. The options are to copy the directory’s contents under a lock the other writers respect, which requires cooperation, or to read the entries through a mechanism that gives a consistent view.

Filesystems with snapshots provide the second option: a snapshot of a directory is immutable, and reading it is a listing of a fixed set of records. That is a filesystem-level feature and not part of the interface. Without it, the closest approximation is to read the whole directory, stat each entry by descriptor-relative open, and treat any entry that fails to resolve as having changed during the scan, which detects the problem without preventing it.

Records left behind by deletion stay in the directory’s data as gaps with inode number zero. The kernel skips them before filling the caller’s buffer, so the caller never sees them, but they still occupy space. That gap is invisible through the interface and is the reason a directory’s reported size can be far larger than the sum of its live entries, the same discrepancy described in Directory Entries as a Name-to-Inode Map.