NFS File Handles and the Server That Keeps No Open Files

A client asks a server for bytes 4096 through 8191 of a file. On a local filesystem that request would begin with a path, and the kernel would walk it component by component until it reached an inode. On the wire the request names neither a path nor a descriptor. It carries a short opaque byte string that the server issued earlier, an offset, and a length, and nothing else. Everything the server needs in order to find the data is inside those bytes.

That substitution is the single design decision from which most of the visible behaviour of network filesystems follows: the survival of a server reboot, the absence of a close operation, the strange hidden files left behind by an unlink, and the ESTALE error that appears when a cached identifier outlives the object it referred to.

What travels on the wire instead of a path

The NFS version 2 file handle is a fixed 32 bytes. Version 3 made it variable length with a maximum of 64 bytes, and version 4 raised the maximum to 128. The protocol never specifies what those bytes mean. A client is required to treat the handle as opaque, to compare handles only for equality, and to store them without interpretation. This is deliberate: it lets a server encode whatever it likes, and it lets the same client code talk to servers with completely different internal layouts.

The read call’s argument structure is therefore almost trivial: handle, offset, count. In version 3 a write adds the data and a stability hint. A lookup takes a directory handle and a single name component and returns a new handle. Notice what is missing. There is no path, no file descriptor, no per-open read position, and no notion of which process issued the call. Two clients reading the same file at different offsets send two independent calls carrying two different offsets, and the server has no reason to connect them.

Removing the path from each request is also a performance decision. Resolving a path costs one directory lookup per component, and on a server with a large directory that means reading directory blocks and comparing names. A handle turns a deep path into a single array lookup. It also removes an ambiguity that a path-based protocol cannot avoid: if another client renames a directory while a request is in flight, a path-based request may resolve to a different object than the one the caller meant, while a handle-based request cannot.

Building a handle the server can trust after a reboot

The interesting engineering is on the server side, because the handle must remain valid across a restart without any in-memory table to consult. The common approach encodes three things into the bytes: a filesystem identifier, an inode number, and a generation counter. The sketch below is schematic — the lookup and reference-count calls stand for whatever a given kernel provides — but the three-field structure and the two checks are the substance.

struct nfs_fh_body {
    uint64_t fsid;        /* which exported filesystem */
    uint64_t ino;         /* inode number within it */
    uint32_t generation;  /* bumped when the inode is reused */
    uint32_t flags;
};

/* Returns 0 on success, ESTALE if the handle no longer denotes a live object. */
int fh_to_inode(const struct nfs_fh_body *fh, struct inode **out)
{
    struct inode *ip = lookup_inode(fh->fsid, fh->ino);

    if (!ip)
        return -ESTALE;
    /* An inode number alone is not an identity: numbers are recycled. */
    if (ip->i_generation != fh->generation) {
        put_inode(ip);
        return -ESTALE;
    }
    *out = ip;
    return 0;
}

The generation counter is what makes the encoding honest. Inode numbers are reused as soon as a file is deleted and its slot is handed to a new file, so a handle built only from a number would silently address the wrong object after that reuse, returning another user’s data instead of an error. Bumping a counter on every reuse, and storing the counter in the inode, turns that silent aliasing into a detectable failure.

A second check follows every operation. The server must confirm that the inode named by the handle still lives underneath the root of the export through which the handle was obtained. Otherwise a client holding a handle from one export could reach files outside it by guessing numbers, or a file moved into a private subtree could remain reachable through an old handle. This ancestry check is the reason exports are described by subtree rather than by device, and it costs a walk up the directory tree on each new handle rather than on each request.

Some servers take the other route and keep a persistent handle table, mapping a random identifier to a stored path or inode, written durably whenever a file is created. That design survives a filesystem change that would break an encoded handle, at the cost of a durable write on the create path and a table that must be rebuilt after a crash. The trade is between cheap handle construction and independence from the underlying layout.

What statelessness removes from the server

Because the server holds no open-file table, open on the client is pure local bookkeeping. The client resolves the path itself, component by component, and remembers the resulting handle in its own structures. No message announces the open, and no message announces the close. In version 2 and version 3 there is no close operation at all.

The absence of an open table has consequences that reach the application. A server cannot enforce share modes, because it does not know which files are open or by whom. It cannot refuse a delete because someone else has the file open. It cannot know that a file is in use and therefore cannot defer a truncation. Advisory locking, which depends on knowing who holds a lock, cannot be expressed in the base protocol, which is why it was specified as a separate service; byte-range locking over the network exists because the file protocol itself had nowhere to put the lock state.

The absence of per-open offsets means readahead is a client concern. The server sees a sequence of independent reads and has no way to know which of them belong to one sequential stream, so it cannot adapt a per-descriptor readahead window. Clients compensate by detecting sequential access themselves and issuing larger reads. Version 2 fixed the data limit at 8 KiB: its read and write calls carry at most 8192 bytes of file data in a single request, so a megabyte of sequential I/O needs at least 128 round trips. Version 3 removed the fixed figure and made the maximum a negotiated property of the mount, obtained from the server and capped by the client’s own limits, which is what allows a client on a fast network to ask for far larger transfers. The change matters mostly because the per-call round trip is what dominates, not because the server was doing anything different per byte.

The payoff arrives when the server restarts. A stateless server that comes back with the same data can answer the retransmitted request as though nothing happened, because the request carried all of its own context. There is no state to lose and no client to notify. The cost of that property is paid by the retransmission layer, which must decide how many times a request may be sent before the client concludes the server is gone rather than merely slow.

The file that has no name

POSIX says that unlinking an open file does not destroy the data; the name goes away and the inode survives until the last descriptor closes. A server with no open table cannot implement that rule, because at the moment of the unlink it has no idea whether anyone holds the file open. Deleting immediately would break readers; refusing the delete would break the caller.

The NFS client resolves this by renaming before it unlinks. It picks a name in the same directory, conventionally with a .nfs prefix followed by a hexadecimal suffix, renames the file to that name, and remembers the handle. When the last local descriptor closes, the client sends a remove for the hidden name. The file stays reachable by handle for exactly as long as a local process needs it, and the directory entry that a subsequent readdir would show is one that no application asked for.

Two failure modes follow. If the client crashes before the final remove, the hidden file is left behind and occupies space until an administrator removes it; directory listings on NFS mounts therefore occasionally contain names beginning with .nfs that no user created. And the rename itself can fail, for instance when the directory is not writable by the client’s credentials, in which case the unlink proceeds immediately and an open reader may begin receiving ESTALE or truncated data. The semantics degrade rather than break, which is the general pattern for a protocol that cannot represent the state the semantics require.

Where the model fails, and why version 4 changed it

ESTALE is the handle’s way of reporting that the identifier no longer denotes anything. It appears when the inode has been freed, when the generation counter no longer matches, when the export has been removed, or when the server has been rebuilt with a different filesystem identifier. A client cannot distinguish these causes; all it knows is that the handle it cached is dead, and its only recovery is to re-resolve the path from a known-good directory handle. If the path has also changed, the application sees the error.

The related hazard is a stale path rather than a stale handle. A client that cached /a/b/c revalidates by walking from the root; if another client renamed b, the walk fails at that component and the cached path is worthless even though the handle still works. Clients therefore treat the path as a hint and the handle as the identity, which is why a file can be renamed by one client and remain open by another without either noticing.

Version 4 abandoned statelessness because two features could not be built on top of it: byte-range locks and cache delegations. Both require the server to know which client holds what, and to be able to recall it. Once the server keeps state, it needs a recovery protocol for the case where that state is lost, which is the subject of grace periods and state reclaim. The handle survived the transition unchanged in spirit, and remains the thing that makes a request self-describing enough to be retried.