Attribute Caching and Close-to-Open Consistency

A program calls stat on a file in a loop, checking whether a background job has finished writing. On a local filesystem each call is a memory read of the inode, and the loop costs nothing. On a network filesystem, doing that faithfully would mean one round trip per iteration, each costing a network hop plus a server-side inode lookup, and the loop would run thousands of times slower than the work it is waiting for.

Caching the answer is the obvious response, and it raises a second problem immediately: what a program sees when another machine changes the file. Local filesystems answer this with cache coherence maintained by hardware or by the kernel, so that every reader sees a change at a defined point. A network client has no such hardware, and a server that pushed invalidations to every client would need to track every client and every cached object. NFS does not attempt that. It caches aggressively and offers a weaker, precisely defined guarantee instead.

The cost structure that forces a cache

The numbers set the design. A local stat on a cached inode is tens of nanoseconds. A network round trip on a local network is a few hundred microseconds, and across a wide area it is tens of milliseconds. Three to six orders of magnitude separate them depending on the network, and a program that performs a few hundred stat calls to walk a directory tree multiplies that difference by the number of files.

The pattern that dominates real workloads is repeated access to the same small set of files. A compiler stats the same headers repeatedly while processing translation units; a build tool walks the same directory looking for changed inputs. Caching attributes for even a few seconds removes almost all of those round trips, because the reuse interval is far shorter than any plausible cache lifetime.

The protocol makes this possible by attaching a validity signal to the attributes themselves. In version 2, the signal was a modification time with one-second resolution. Version 3 kept timestamps but widened them: the attributes carry separate access, modify, and change times, each with a seconds field and a nanoseconds field, so two writes a microsecond apart are distinguishable on the wire. What neither version provides is a counter. Version 4 added one: the change attribute, an unsigned integer the server bumps whenever the inode’s metadata changes, which lets a client settle the whole question with a single field comparison instead of comparing clocks.

What a client does with a cached entry

The client keeps a timestamp of when it last validated each cached object, and a lifetime chosen from a small set of rules. When an application asks for attributes, the client compares the current time against that lifetime.

If the entry is still fresh, the cached values are returned with no network traffic. This is the common path, and it is what makes directory walks and repeated stat calls usable.

If the entry is stale, the client sends a get-attributes call and compares what comes back against what it holds. Under a version 4 change attribute the test is exact equality. Under version 3 the client has only timestamps, so it compares the modification and change times against those it recorded when the cached data arrived; equal times are taken to mean the file is unchanged. If the comparison says unchanged, the client refreshes its validation timestamp and keeps the rest of the cached data — including any cached file pages, which is the important part, since re-fetching a large file because its metadata was re-checked would defeat the purpose. If the times differ, the cached pages are dropped and the new attributes replace the old.

A third case exists for objects the client is actively modifying. While a file is open for writing, the client is the authority on its own data, so it serves reads from its own page cache without consulting the server. This is safe only because of the rule described below, which constrains what other clients are allowed to assume.

/* Schematic validation decision: the shape, not a kernel interface. */
if (now < entry->valid_until)
    return entry;                     /* fresh: no traffic at all */

attr = nfs_getattr(handle);           /* one round trip */
if (same_version(attr, entry)) {      /* v4: change attr; v3: mtime+ctime */
    entry->valid_until = now + ttl;   /* unchanged: extend, keep pages */
    return entry;
}
invalidate_pages(entry);              /* changed: drop cached data */
entry->attr = attr;
entry->valid_until = now + ttl;
return entry;

The lifetime itself is not fixed. Each class of object has a lower and an upper bound: a freshly validated entry is trusted for at least the lower bound, and once it is older than the upper bound it is revalidated rather than served. The Linux client exposes the two bounds per object class as mount options, applied separately to regular files and to directories, and the shipped defaults are on the order of a minute: acregmax, acdirmax, acregmin, and acdirmin in nfs(5) default to 60 seconds, 60 seconds, 3 seconds, and 30 seconds respectively. Those are conservative relative to correctness rather than relative to traffic, because the cost of a wrong answer is a program misbehaving rather than an error being reported.

Close-to-open: the guarantee that is actually offered

The consistency model that NFS documents is close-to-open. It has two halves, and the name states both.

When a client opens a file, it checks with the server that its cached copy is still current, dropping cached data if the server’s metadata has moved past what the client holds. The open path therefore acts as a barrier: whatever a previous writer flushed before its close is visible to this reader. This is why a program that writes a file on one machine and then starts a reader on another usually sees the new contents — the reader’s open triggers the validation.

When a client closes a file, it flushes any dirty data to the server before the close returns. The close is the point at which other machines become entitled to see the writes. A program that relies on “the data is visible elsewhere as soon as write returns” is relying on something NFS never promised.

Between open and close there is no guarantee at all about what another client sees. Two clients writing the same file concurrently will interleave in whatever order their writes reach the server, with no ordering between them and no notification to either. A reader on a third client may see a mixture of both writers’ data, or the older version, depending on which cached pages it holds and when it last validated.

The gap is deliberate. Strong coherence would require the server to know, per client and per object, which cached pages exist, and to send invalidations when they change. That means a callback channel, per-client state, and a recovery protocol for when a callback cannot be delivered. Delegations build exactly that machinery, but they do so for the narrow case of a client that is the sole opener, and they revoke the delegation as soon as a second client appears precisely because maintaining the guarantee for concurrent access is expensive.

Where the model produces surprising behaviour

The classic failure is a program that writes a file and immediately reads it back through a different path, expecting to see its own writes. With close-to-open, a client that has the file open for writing serves its own reads from cache, so this works within one client. Across clients it works only if the writer closed first.

The second surprise is timestamp granularity. Version 2 carried one-second timestamps, so a file written twice within the same second looked unchanged to a client comparing times, and build tools that use modification times as their only signal could skip work. Version 3 added nanosecond fields, which narrows the window but leaves the underlying problem: change detection compares a server-reported time against a previously reported time from the same server, and two changes that land in the same timestamp — or a server whose clock is set backward — make the comparison lie. Version 4’s change attribute exists to avoid that comparison entirely, since it is a counter rather than a clock and only ever moves forward.

A third issue is negative caching. Looking up a name that does not exist costs a round trip, and programs that probe for optional files do so constantly. Clients cache the absence for a short period, so a file created by another machine may remain invisible for the length of that negative entry. The window is kept short — shorter than positive attribute lifetimes — because the failure it causes is a program concluding that a file does not exist when it does.

The fourth is the interaction with O_DIRECT and with memory-mapped access. Mapped pages follow the same validation rules as read pages, but a mapping has no open and no close in the sense the model uses, so the validation points are the page faults rather than the open call. A long-lived mapping on a client therefore holds pages that another client has since replaced, and the local process can keep reading the old contents indefinitely. This is not a defect in the implementation; it is what the model permits, and it is why programs that share files across machines and depend on immediate visibility use locking rather than assuming coherence.

Why not simply make the cache short

Shortening the lifetimes approaches coherence from the wrong direction. A lifetime of zero means a round trip per access, which is the cost the cache exists to avoid. A lifetime of one second still leaves a window, and the window is not what protects correctness — the open-time validation is. The lifetime controls how much traffic the client generates, and the validation rule controls what the client is allowed to conclude.

The practical consequence is that the model is tunable along one axis: shorter lifetimes buy fresher answers at the cost of more traffic, and longer lifetimes buy fewer round trips at the cost of longer windows in which two clients disagree. Neither setting makes concurrent writers safe. That requires byte-range locking, which serialises access at the granularity the application chooses, and which is the only mechanism in the protocol family that lets a client assert exclusive access rather than hope for it.