Grace Periods and Reclaiming State After a Server Restart

Version 2 and version 3 of NFS kept no state on the server, so a restart was invisible. The server came back, the data was still there, and a client that retransmitted a request received an answer. Nothing needed to be recovered because nothing had been remembered.

Version 4 gave up that property to support locks and delegations, and in doing so acquired a problem the earlier protocol did not have. A server that reboots loses its table of opens, locks, and delegations. Its clients do not know this. They continue to send operations naming state identifiers that the server no longer recognises, and they continue to believe they hold exclusive access to ranges that are now unprotected. The protocol needs a defined interval during which clients can restore what they had before anyone else is allowed to compete for it.

The window between losing state and trusting clients again

The naive approach after a restart is to accept whatever clients ask for. That fails immediately. Client A held an exclusive lock on bytes 0 to 4095 and is still writing under that assumption. Client B, unaware of the restart, asks for the same range and is granted it. Both now write to the same range, and neither is told.

The opposite approach — refuse everything until clients prove their state — is unimplementable, because there is no proof. The server has no record to compare against. Any client can claim any set of locks.

What the protocol does instead is partition time. For a defined interval after a restart, the server enters a grace period. During it, a request that carries a reclaim flag is treated as a restoration of prior state and granted subject to conflicts with other reclaims. A request without the flag is rejected with a specific error meaning “try again after the grace period”. New work is deferred; restoration proceeds. After the interval ends, the server accepts normal requests, and anything not reclaimed is gone.

The ordering guarantee this provides is the important part. Every client that had state has the same window in which to restore it, and none of them can be preempted by a client that is starting fresh. A client that held a lock and reclaims it inside the window gets it back, even if another client would have wanted it.

Reclaims are not re-arbitrated as ordinary requests would be, and that is the weakness of the scheme. The server has no record of what a client held, so it cannot tell a genuine reclaim from a claim on a range nobody held. RFC 7530 states the split responsibility plainly: a client must not reclaim a lock it did not hold at the end of its most recent established lease, and a server must not accept a reclaim unless it knows it could not have granted a conflicting lock in the meantime — while being permitted to assume clients are honest in making that judgement. The specification’s own worked example shows the consequence: a client whose lease expired during a partition, whose lock was released and taken by another client, and which then reclaims after a reboot is granted the lock on state that was never continuously held. The error NFS4ERR_RECLAIM_CONFLICT exists for the case where a reclaim does collide with another client’s live lock, and the specification describes that outcome as possible only when a client has misbehaved.

How a client knows it must reclaim

The trigger is a failure the client can recognise as a restart rather than a transient loss. Three signals are available, and implementations use them in combination.

The first is the transport. A TCP connection that is closed and cannot be re-established to the same server indicates the server is gone. When the connection returns, the client knows a new server instance is answering, though not necessarily that it restarted.

The second is an explicit state error. The server answers an operation with an error indicating that the state identifier is unknown. This is definitive: the state the client held no longer exists. What the client can do about it depends on why it went away — state revoked because the client’s lease expired cannot be reclaimed, while state lost to a server restart can be, if the grace period is still open.

The third is the server’s own announcement. The protocol has no query that asks a server whether it has rebooted, but the answer surfaces in the identification exchange. A version 4.0 client that presents a client identifier the server no longer recognises is told so with a stale-client error, which means the server no longer holds a record of that client. In version 4.1 the exchange of client identity returns a flag, EXCHGID4_FLAG_CONFIRMED_R, indicating that the server already had a confirmed record for this client, from which the client derives whether it is talking to an instance that knew it before. A client that learns the server is in grace reclaims rather than opening fresh.

The signals are not equally reliable. A client that has been partitioned for a long time may reconnect after the grace period has already expired, in which case its reclaim is rejected and it must treat its state as lost. A client whose connection was broken by an intermediary may reconnect to the same server instance and find its state intact, in which case reclaiming would be wrong — the server would see a duplicate. The protocol handles this by having the reclaim be idempotent: reclaiming state that already exists succeeds without creating a second copy.

/* Schematic: the client's decision after the server loses its records. */
if (err == NFS4ERR_STALE_CLIENTID) {
    /* Our state is gone. Reclaim within the grace window. */
    establish_client_id();
    rc = send_reclaim_open(file);        /* CLAIM_PREVIOUS */
    if (rc == NFS4ERR_NO_GRACE || rc == NFS4ERR_RECLAIM_BAD)
        reopen_from_scratch(file);       /* window closed: state lost */
}

The NO_GRACE outcome is the one that costs data. The client’s lock is gone, other clients may hold the range, and the writes the client was performing under the assumption of exclusion were unprotected. There is no recovery from this at the protocol level; the application’s own consistency checks are the only defence.

Sizing the grace period

The interval is bounded from below by how long it takes the slowest client to notice the restart and send its reclaims. That depends on the client’s retry configuration, on the number of clients, and on how many pieces of state each must restore.

A client with a long retry interval may not attempt a reconnect for tens of seconds. A client with thousands of opens must reclaim each one, and if the server processes them serially the total time grows with the count. A site with many clients reclaiming at once creates a burst of load on a server that has just started, which slows the reclaims further.

The protocol sets a floor rather than a default. Where the grace period follows a migration or a reboot whose prior lease is known, RFC 7530 requires the new server’s grace period to be at least as long as the previous lease_time attribute, because a client is entitled to a reclaim window measured in the lease it was working under. Otherwise the duration is a server attribute that the administrator sets, and it should be set from the observed behaviour of the client population rather than from a specification constant. The cost of a longer interval is that new work is refused for longer. The cost of a shorter one is that some clients lose their state silently.

There is a middle path some implementations take: end the grace period early once every known client has either reclaimed or been determined to be gone. That requires the server to know its client set, which version 4 provides through client identifiers and the associated session or lease records. A server that has heard from all of its known clients can stop waiting, since no further reclaims are possible from clients it knows about. The risk is a client the server has forgotten — one whose record was lost in the same crash — which would then be refused and have to start fresh.

Leases and the second clock

Running alongside the grace period is the lease. A version 4 client holds its state under a lease, a period during which the server promises to keep that state and to honour the client’s cached data. The client renews the lease by sending operations; a client that goes silent for longer than the lease duration has its state revoked.

Version 4.1 replaced the single lease with sessions, each having a lease time and a set of slots. The session’s lease is renewed by operations on the session, and a session that is not renewed is destroyed, releasing its state. This makes the bookkeeping more explicit: the server knows exactly which sessions exist, can enumerate them, and can report their state. It also makes the grace period more tractable, because the set of clients that might reclaim is the set of sessions the server had recorded.

The lease duration is the parameter that governs how quickly the server can reclaim resources from a dead client. Too short and a client that is merely busy loses its state; too long and a crashed client’s locks stay held for the whole duration. RFC 7530 specifies the attribute’s meaning and unit — a duration in seconds — and deliberately no value, because the right number depends on the network and on how many clients a server holds state for. A client renews well inside the interval rather than at its edge, so that one delayed renewal does not cost it everything it holds.

What recovery cannot restore

Reclaim restores state, not data. A server restart does not touch the client’s page cache: the pages that had not reached the server still exist on the client, and they are simply data that was never made durable. The danger is on the server’s side, because whatever the server acknowledged as stable is gone if it did not survive the crash, and whatever the server held as unstable is gone too. The commit protocol exists to bound how much can be lost, and the verifier exists to tell the client when a commit no longer covers what it thinks it covers. After a restart the server’s verifier changes, so a client that tries to commit writes from before the crash receives a mismatch and must resend the data — which it can, because the pages are still in its cache. The loss is of the server’s word about durability, not of the client’s copy of the bytes.

The second thing recovery cannot restore is the ordering between clients. Suppose two clients each held a read lock on a shared range and one was writing under a separate write lock elsewhere. After the restart both reclaim, both succeed, and the interleaving that occurs is whatever the reclaim order produces. The locks are restored; the dynamic behaviour around them is not.

The third is the identity of a client. A client that has rebooted loses its client identifier unless it stores it, and a client identifier that is reused by a different machine is a security problem rather than a performance one — the new machine would inherit the old one’s opens. The protocol requires the client to prove continuity, typically by presenting a verifier derived from stable local state, and a client that cannot do so is treated as new. This is why a client reboot is a heavier event than a server reboot from the client’s point of view: it must discard its own state rather than reclaim it.

The fourth is the state of a client that was in the middle of a recall when the server died. A delegation that was being recalled is neither held nor returned. After the restart the server may have a client reclaiming the delegation and another client waiting for the conflicting open. Whether the reclaim succeeds depends on whether the conflicting request was itself recorded, and the protocol’s answer is that the waiting client retries and the resolution happens after the grace period.

Why the design is a bet rather than a guarantee

Every parameter here — grace duration, lease duration, retry interval, reclaim idempotency — encodes an assumption about client behaviour under failure. The server assumes clients will notice within the grace period. The client assumes the server will honour a reclaim it accepted. Neither assumption is enforceable, and the protocol is written so that violations degrade rather than corrupt: a late reclaim is refused, a revoked lease produces a state error, a lost delegation falls back to per-request validation.

The degradation is not always graceful. A client that loses a write lock and continues writing will produce data that another client also believes it owns, and the resulting file may be internally inconsistent in a way no protocol can detect. This is the price of stateful caching, and it is why the stateless design described in NFS file handles and statelessness remains attractive for filesystems where the server’s availability cannot be guaranteed. The stateful design buys performance and correctness for cooperating clients, and it pays for them with a recovery protocol whose failure modes are worse than the stateless model’s.