pNFS Layouts: Separating the Metadata Path from the Data Path

A single NFS server answering every read and write for a large dataset has one network interface, one set of disks, and one CPU, and the throughput ceiling is whatever that combination can deliver. Adding clients does not help: they all queue at the same place. The parallel extensions to NFS version 4.1 address this by letting the client bypass the server for bulk data, while the server retains control over where the data lives and who may touch it.

The mechanism is the layout: a grant, obtained by asking the server, that describes where the file’s bytes are and what the client is permitted to do with them directly. The server is still the metadata authority. It is no longer necessarily the data path.

Why the metadata server keeps its role

Splitting the roles is not the same as removing the server. The metadata server still owns the file’s namespace, its attributes, its permissions, and its layout assignments. A client that wants to read a file must first obtain a layout, and the layout is issued under the same state model as any other NFS operation — it belongs to an open, it has a state identifier, it can be recalled.

The reason the server cannot simply hand out storage addresses and forget about them is that the layout is a promise about consistency. If the client writes directly to a storage device, the metadata server’s view of the file’s size and modification time becomes stale the moment the write lands. The protocol handles this by requiring the client to report changes back: the client must tell the server when it has written, so the server can update attributes and, if necessary, invalidate other clients’ layouts.

The division of labour is therefore asymmetric. Data flows between client and storage device without passing through the server. Control — layout grants, recalls, and commit notifications — flows between client and server. The server’s throughput is no longer the bottleneck for data, but it remains the serialisation point for metadata, which means workloads dominated by small-file operations gain much less than workloads dominated by large sequential transfers.

The layout, as a data structure

A layout is returned by a LAYOUTGET operation, which names the file, the byte range, the layout type the client prefers, and the access the client intends — read, write, or both. The server’s reply carries a layout type, an opaque layout body whose interpretation depends on the type, a state identifier, and a length indicating how much of the file the layout covers.

Three layout types were defined in the original specification. The file layout describes the file as striped across a set of data servers, each with its own NFS file handle and a set of stripes; the client reads and writes each stripe with ordinary NFS operations against the appropriate data server. The block layout describes the file as volumes and block ranges, and the client accesses them with SCSI commands over a storage network, which requires the client to have a device address and a means of issuing I/O to it. The object layout describes the file as a set of objects on object storage devices, accessed with a small protocol whose operations are expressed in the layout.

The common structure across all three is the same: a list of storage locations, a mapping from file offsets to locations, and a set of constraints on what the client may do. The file layout’s body also carries a flag controlling how stripe units are packed into the data server’s file. Dense packing stores each server’s units contiguously; sparse packing leaves a hole where another server’s units would fall, so that a file’s stripes keep their offsets inside every data server’s file. Sparse packing therefore requires every data server to store a file with holes, which is one reason a layout may specify dense packing instead; an I/O to one of those holes is refused with a specific error rather than fetched from somewhere else.

/* Schematic: the shape of the decision, not a library interface. */
struct layout *lo = layout_get(file, 0, whole_file, layout_file, access_rw);

if (lo == NULL)
    return read_via_server(file);          /* no layout: fall back */

/* Offsets outside the granted range are read through the server. */
if (off + len > lo->length)
    return read_via_server_range(file, off, len);

struct stripe *s = stripe_for_offset(lo, off);

if (s == NULL)
    return -NFS4ERR_PNFS_IO_HOLE;          /* sparse mapping, no unit here */

return read_from_data_server(s->ds, s->fh, s->offset_in_stripe, len);

The fallback path is not a degraded mode invented by the client; it is part of the design. A layout may cover only part of a file, and a client is expected to use the server for the rest. This lets a server issue layouts where it has storage to spare and keep the rest on its own disks. The two cases the code separates are different in kind: an offset beyond the layout’s range is simply not covered and must go to the metadata server, whereas a sparse hole is a gap inside the granted range and is an error the client must surface.

The three layout types and the cost of each

The file layout is the simplest to deploy because it reuses everything NFS already has. Data servers are ordinary NFS servers holding file handles, and the client’s I/O is ordinary NFS I/O, just addressed to a different server. The cost is that the client still speaks the full NFS protocol per stripe, which means per-request overhead and per-request credential checks. It buys parallelism across servers rather than a cheaper protocol.

The block layout removes the protocol overhead entirely by letting the client issue SCSI commands to storage. A read becomes a device read rather than a network filesystem request, which is the fastest of the three. The cost is infrastructural: the client must have access to the storage network, must be able to address the devices, and must have credentials for them. The metadata server hands out volume identifiers and block ranges, and the client is responsible for the mapping from file offset to logical block address. This is the layout type that requires the most from the client’s environment and delivers the most when that environment exists.

The object layout sits between them. Objects are addressed by identifier and accessed with a protocol designed for storage devices rather than for filesystems, so the client avoids filesystem semantics per request while not needing raw block access. The metadata server hands out object identifiers and a mapping, and the client issues reads and writes against object storage devices.

The choice of layout type is a deployment decision, not a client preference the server must honour. A client asks for the type it supports; a server that does not offer it returns an error, and the client falls back to ordinary NFS access. This is why a filesystem can support several layout types simultaneously and why a client’s capabilities determine what it gets.

Recall, and why layouts must be revocable

A layout grants authority, and authority must be recoverable. The case that forces it: a client holds a write layout covering bytes 0 through 1048575 and has been writing directly to storage. A second client opens the same file for reading. The server cannot answer that read from its own view of the file, because the first client’s writes went around it.

The server recalls the layout, using the same callback channel that carries delegation recalls. The client must stop writing directly, flush what it has written, and report the file’s new size and modification time to the server. Only after that can the server answer the second client with correct attributes.

The recall has a defined sequence. The server sends a layout recall naming the file and the layout. The client, on receiving it, completes any in-flight I/O to the data servers, sends the server a commit-like notification of what it wrote, and returns the layout. The server then invalidates the old layout and can issue a new one.

If the client does not respond, the server can revoke the layout after a timeout, in the same way it can revoke a delegation. The difference is the consequence: a revoked write layout means the client’s direct writes are no longer coordinated with the server’s metadata, and the file’s attributes may be permanently wrong until something forces a resynchronisation. This is a more severe failure than a revoked read delegation, and it is why write layouts are issued more conservatively.

Layouts interact with a server restart in a way that differs from delegations, which is where they meet the grace period and state reclaim machinery. A client that reclaims its open after a restart cannot reclaim its layout. Asking for a layout again is an ordinary request, and the answer may name different storage devices, a different range, or a different layout type, so the protocol gives the client no way to say “give me back exactly what I had”. What the client can do is report what it wrote under the old layout, using a commit operation whose reclaim flag tells the server that the reported writes predate the restart even though the client no longer holds the layout. That report reconciles the data servers with the metadata server’s view of the file, and it deliberately grants nothing: the client then obtains a fresh layout when the grace period ends, or falls back to server-side I/O if no new layout is available.

What the client must track that ordinary NFS never needed

An ordinary NFS client tracks open files and cached pages. A pNFS client additionally tracks, per file, the layouts it holds, the byte ranges they cover, the access each grants, and the state identifiers that make them valid. It must also track which pages were written through which layout, because the notification to the server must describe the writes that went around it.

The bookkeeping creates a failure mode that does not exist without layouts: a page can be dirty in the client’s cache with no record of which data path it was destined for. If a layout is recalled while such a page exists, the client must either write the page through the server or through the data server before returning the layout, and choosing wrong produces a write that the server does not know about.

There is also an interaction with the client’s own caching model. The close-to-open rule says that a close must make writes visible to other clients. With a write layout, “visible” means the server’s metadata must reflect the writes, which requires the notification step. A close therefore cannot return until the client has told the server what it wrote, even though the data itself never touched the server.

Where the design does not help

Layouts improve throughput for large files accessed by clients that can reach the storage directly. They do nothing for metadata-heavy workloads, because every lookup, create, and rename still goes to the metadata server. They do nothing for small files, because the layout overhead — a request, a grant, a mapping computation, and a notification on close — exceeds the cost of simply reading the file through the server.

They also complicate the failure model. A storage device that becomes unreachable is a failure the metadata server may not know about, and the client must decide whether to retry the device or fall back to the server. The server, meanwhile, may have issued the same layout to other clients and has no way to learn that the device is down except from their errors. The protocol provides error reporting, but the coordination it enables is weaker than the coordination a single server has over its own disks.

The final limit is that a layout is a snapshot of an assignment. The server can move data between storage devices, and when it does, the layouts that described the old placement become wrong. The protocol requires the server to recall those layouts before moving the data, which means a rebalance is a coordinated operation rather than a background one. That constraint keeps the design correct and makes storage rebalancing a visible event, which is the trade the parallel extension accepts in exchange for letting clients read and write at the speed of the storage rather than at the speed of the server.