Unstable Writes, COMMIT, and the Verifier

Version 2 of NFS had one write operation and one implicit promise: when the reply arrived, the data was on disk. That promise is expensive. It forces the server to wait for a physical write before answering, and if the client is sending many small writes the server performs a synchronous disk operation per request, which is the classic reason early NFS servers were slow on write-heavy workloads.

Version 3 broke that single promise into three, and made the choice explicit. A write reply now carries a stability indicator, and the client can ask for a specific level. The result is a protocol where the client decides, per write, whether to pay for durability now or defer it — and where deferring requires a later call to make the guarantee true.

Three levels of stability, and what each costs

UNSTABLE means the server has accepted the data and may still hold it in volatile memory. The client must not assume the write survives a server crash. DATA_SYNC means the file’s data is on stable storage, though its metadata may not be. FILE_SYNC means data and metadata are both durable, and the file is consistent after a crash.

The server is permitted to reply with a higher level than requested, and a client must accept that. A server that has already flushed because of an unrelated sync, or one whose filesystem journals writes in batches, may find that the data is durable by the time the reply is composed. The reply’s field describes what actually happened, not what was asked for.

The asymmetry is deliberate: a client may ask for less stability than it needs only if it intends to commit later, and a server may deliver more without harm. This makes the protocol tolerant of servers with different write policies and lets a client’s behaviour be correct against all of them.

/* Schematic: what the client must track after an unstable reply. */
struct wlist {
    struct list_head link;  /* node on the file's unstable-range list */
    uint64_t offset;
    uint64_t count;
    uint8_t  verifier[8];   /* opaque write epoch, eight bytes on the wire */
};

/* After a write reply with stable == UNSTABLE, record the range. */
void note_unstable(struct file *f, uint64_t offset, uint64_t count,
                   const uint8_t verifier[8])
{
    struct wlist *w = kmalloc(sizeof(*w), GFP_KERNEL);

    w->offset = offset;
    w->count = count;
    memcpy(w->verifier, verifier, sizeof w->verifier);
    f->dirty_bytes += count;
    list_add_tail(&w->link, &f->unstable);

    /* Commit eagerly once the uncommitted set is large. */
    if (f->dirty_bytes >= f->commit_threshold)
        send_commit(f->handle, 0, 0);   /* offset 0, count 0: everything */
}

The commit call takes an offset and a count, so a client can commit a specific range, and a count of zero means “all unstable writes for this file”. The write reply carries an eight-byte verifier alongside the stability level, and the commit reply carries one of its own. The client compares them against what it recorded: a verifier that differs tells the client that the server’s write epoch changed while its data was in flight, so the range it thinks is pending is not the range the server holds. The stability level works the other way round from what a caller might expect: the client asks for a minimum, and the server is allowed to reply with a higher level — a server that had already flushed the data can answer FILE_SYNC to a request that only asked for UNSTABLE, and the client accepts it. What the server must not do is claim FILE_SYNC and then behave as if the data were volatile, which is the state the protocol’s commitment field is meant to make impossible to misreport.

Why a verifier is not a checksum

The verifier is an opaque 64-bit value chosen by the server. It is not a hash of the data, and the protocol does not require it to change when the data changes. What it identifies is a write epoch on the server: the set of writes that the server will flush together, or that belong to one incarnation of the file.

The distinction matters when the server restarts. Suppose a client wrote a megabyte with UNSTABLE and then sent a commit. The server flushed and replied. Now suppose the server crashes and comes back, losing whatever volatile state it had. A commit from before the crash cannot be honoured, because the server no longer knows which writes it covered. The verifier lets the server report this: after a restart it uses a new verifier value, so a client that compares its recorded verifier against the reply’s sees a mismatch and knows its earlier writes are not covered.

A checksum would not help here. The server could recompute a checksum of the data it holds, but the question is not whether the bytes are intact — it is whether those bytes are the ones the client wrote. Only the server’s own record of the write can answer that, which is what the verifier stands in for.

The other use of the verifier is to detect a replay problem. If a client retransmits a write and the duplicate request cache has no entry, the server executes it again. A commit whose verifier does not match the current epoch tells the client that the state it assumed has changed, so it should reissue the writes rather than assume the data is safe.

The trade the client actually makes

Consider a client writing a large file sequentially. With FILE_SYNC on every write, each write waits for a disk flush, so throughput is bounded by the server’s synchronous write rate rather than by the network. With UNSTABLE writes followed by periodic commits, the client streams data at network speed and pays for a flush once per batch.

The batch size is the tuning knob, and it has a correctness dimension. Every byte written with UNSTABLE and not yet committed is at risk if the server fails. The client’s own data structures may say the write completed, and the application may have been told so. If the server then crashes and the client learns that the writes were lost, the application has been lied to. So the client must hold on to enough information to retry those writes, which means keeping the data or being able to regenerate it.

That constraint is what limits the batch. A client that writes a gigabyte of unstable data must either keep a gigabyte of dirty pages available for retransmission or accept that it cannot recover them. Real clients bound the uncommitted set by memory and by time, and force a commit when either limit is reached. The threshold is a policy choice: larger batches amortise the flush cost, smaller batches bound the amount of state that must be retained.

There is a second-order effect on the server. Unstable writes let the server accumulate dirty data and flush it in whatever order suits the disk — sorted by block address, for instance, which is much faster on rotating media than flushing in arrival order. This is why unstable writes are not merely a way to shift latency; they can improve the server’s throughput, since the server gains the freedom to batch and reorder that a synchronous write forbids.

Where the guarantee becomes visible to applications

The stability level is not something an ordinary application sets. It is negotiated by the client based on the mount options and on the semantics of the call that triggered the write.

A write system call that returns has been accepted by the local kernel into the page cache. Nothing about the network has happened yet. The dirty pages are sent later by the writeback machinery, with whatever stability the client’s policy chooses. So the guarantee at the write boundary is purely local.

fsync is the call that reaches the protocol. It flushes the file’s dirty pages to the server — sending writes if the data has not left the client yet — and then commits whatever the server reported as unstable, and it waits for the commit reply before returning. The commit is always a round trip of its own; the write phase is a round trip only when there was data that had not yet been sent. This is why fsync on a network filesystem can cost substantially more than on a local disk: the flush has to complete on the server and the commit has to be acknowledged, and both are on the caller’s critical path.

O_SYNC and O_DSYNC make every write synchronous, and on a network mount the natural way to honour that is to request FILE_SYNC or DATA_SYNC on each write so the server does the flush before replying. That keeps the operation to a single round trip per write, but it serialises the stream, because each write must be acknowledged as durable before the next is issued. The cost is severe enough that applications which need durability are usually better served by batching their own writes and calling fsync at meaningful boundaries than by asking the filesystem to make every write synchronous.

The close-to-open model described in attribute caching and validation supplies one more implicit commit point. The rule that a writer’s data is visible to a later opener on another client is a client-side convention rather than a protocol operation: it works because the writing client flushes at close and the reading client revalidates at open. Implementing the first half means a close cannot return until the dirty pages have reached the server and anything unstable has been committed, so closing a heavily written file costs roughly what fsync costs. A client that declined to flush at close would still be speaking the protocol correctly, and would break the guarantee the protocol is usually described as providing.

Failure modes the protocol cannot remove

The commit is a promise about the server’s storage, and it inherits every weakness of that storage. A server that reports FILE_SYNC after writing to a disk with a volatile write cache, and that disk then loses power, has made a promise it could not keep. The protocol has no way to detect this; the client believes the reply and the application believes the client. This is a property of the server’s hardware configuration rather than of NFS: the stability levels describe what the server’s storage stack reports, and a disk whose write cache is not flushed on command invalidates all three of them at once.

A commit that covers a range can be answered while writes outside that range remain unstable. A client that commits bytes 0 through 65535 and then assumes the whole file is durable is wrong, and the protocol’s range argument exists to make that explicit. The common client behaviour of committing offset 0 with count 0 avoids the issue by covering everything, at the cost of flushing data the caller did not ask about.

The commit reply’s verifier can be a source of spurious retries. If the server changes its verifier for a reason unrelated to the client’s writes — a restart, a re-export, a change in the backing store — the client sees a mismatch and must resend. That is the safe direction, since resending is idempotent for a write at a fixed offset, but it means a server restart during a large write forces the client to redo work it had already completed.

Finally, there is a window between a write’s unstable reply and the commit in which the client has told the application the data was written. Programs that write, close, and immediately exit without an explicit fsync are relying on close to commit, which a client implementing close-to-open does; programs that write and then exit without closing, or that rely on the kernel to flush after the process is gone, are relying on the writeback machinery to commit on its own schedule. The risk lives in the interval before the commit completes, and the protocol’s contribution is to make that interval explicit and bounded rather than hidden.