Delegations, Callback Channels, and Recall

A single client opens a file, reads it, and does not share it with anyone. Every read it performs goes to the server, and every read is answered from the same pages. The server knows that no one else has the file open, so it could simply tell the client to stop asking — cache freely, and mention it later if anything changes.

That is a delegation: a temporary transfer of authority over one file from the server to one client. The client stops sending requests for that file, and the server takes on an obligation. If another client wants the file, the server can no longer answer it directly, because the first client has data the second must not miss. The server must reach back to the client that holds the delegation and ask for it to be returned. That reverse direction — the server calling the client, using the client as a server for one small procedure set — is the callback channel, and it is the part of the design that creates most of the complexity.

What a delegation actually grants

A read delegation tells the client that it may serve reads and attribute queries from its own cache without contacting the server, and that the server will notify it before anything changes. A write delegation goes further: the client may buffer writes locally and treat its cached copy as authoritative, which lets it respond to reads of its own recent writes without any network traffic and without the commitment that the unstable write and commit protocol would otherwise impose for visibility to others.

The grant is tied to an open. The delegation is associated with the stateid that the client received when it opened the file, and it is valid only for as long as that open exists. Closing the file returns the delegation implicitly. This keeps the bookkeeping to a single object: there is no separate table of delegations to manage beyond the open state the protocol already requires.

Delegations are not granted when the server believes the file is shared. If another client has an open on the same file, or has a conflicting lock, the grant is withheld or the existing delegation is recalled and the second open is answered normally. The delegation is thus an optimisation that applies to the common case of private access, and its value falls off sharply as soon as a file is genuinely shared.

Two properties make the optimisation worthwhile. The first is traffic: a client with a read delegation on a large file eliminates every attribute check and every cache revalidation for that file, which matters especially for programs that read a file repeatedly or stat it in a loop. The second is latency: writes with a write delegation can be acknowledged locally, so a program that writes a file and reads it back sees no round trips at all. The gain is real but conditional, and the conditions are exactly the ones the callback channel exists to enforce.

The callback channel, which runs the other way

Ordinary call sharing does not work for server-to-client calls, because the server’s address is known and the client’s is behind whatever the network topology provides. Version 4 handles this by having the client register callback information when it establishes its client identity or later creates a session. The record contains an address and a protocol the server can use, and the server confirms the path before relying on it: version 4.0 sends a null callback procedure, which does nothing but prove that a connection can be opened to the registered address, and version 4.1 binds the callback direction to an existing connection with an explicit session operation instead of registering an address for the server to dial.

That the callback path may be unusable is expected rather than exceptional. A client behind a firewall that permits outbound connections and blocks inbound ones can reach the server but cannot be reached. Version 4 accounts for this by treating the callback path as an optional optimisation with defined fallbacks, and version 4.1 goes further by folding callbacks into the session’s back channel, which is normally carried on the same transport as the forward operations — reversing the direction on an established connection rather than requiring a fresh inbound connection.

Because the channel is unidirectional in its purpose, the callback protocol is a small, fixed set of procedures rather than a second file protocol. Version 4.0 defines four: a null call for reachability, a compound wrapper, an attribute query used to detect changes to a delegated file, and the delegation recall. Version 4.1 keeps those and adds operations that only make sense once the server tracks many clients and layouts: the recall of a specific layout, the recall of any delegation or layout matching a class, notification that a recallable object has become available, a reduction in the number of session slots the client may use, cancellation of a pending want, notification that a lock is now free, notification of a device identifier change, a general notification mechanism, and the offer of a delegation to a client whose earlier request for one was denied. Each exists because some server-side decision needs to reach a client that is not currently talking to it.

Failures on this path are handled by retrying the callback and, if the client stays silent, by revoking the delegation after a configured interval. Revocation is the heavy hammer: the server declares the delegation gone, answers the pending conflicting request, and any subsequent use of the revoked stateid by the client fails with a state error. The client discovers the loss when an operation fails, which is the same discovery model that lock recovery uses.

The recall sequence, step by step

The sequence begins when a second client requests access that conflicts with an outstanding delegation. The server cannot simply grant the request, because the delegating client may hold cached data the request must reflect.

1. The server sends a recall for the delegation, naming the file and the stateid.
2. The client stops using the delegation, flushes any dirty data, and returns it.
3. The server completes the conflicting request with the returned delegation no longer protecting the first client’s cache.

The client is expected to respond to the recall as quickly as it can. While the recall is outstanding, the server holds the second client’s request. In version 4.0, servers were permitted to wait for the recall before replying, and a client that was slow or unreachable could stall an unrelated client’s open call for as long as the server was willing to wait. Later revisions tighten this by putting an explicit bound on how long the recall may remain outstanding before the server proceeds, so that the wait is measured in a configured interval rather than in the delegating client’s responsiveness.

The difference matters because the stall is unbounded from the second client’s perspective. Its open call has no timeout of its own beyond the transport’s, and a server holding the request can hold it for as long as it believes the recall will succeed. Moving the delay onto the server’s side of the decision, with a defined revocation point, converts an open-ended wait into a bounded one.

/* Schematic: the sequence a recall handler must follow. */
int handle_recall(struct stateid *sid)
{
    struct open_state *st = find_state(sid);

    if (!st)
        return -NFS4ERR_BAD_STATEID;   /* already returned or revoked */

    flush_and_wait_dirty_pages(st->file);  /* data must reach the server */
    if (st->mode == WRITE)
        send_commit(st->handle, 0, 0);     /* all unstable writes */

    /* Return the delegation but keep the open: the app still has it. */
    return send_delegreturn(st->handle, st->delegation_sid);
}

The subtlety in that sequence is the difference between returning a delegation and closing a file. The application still holds the file open and expects reads and writes to keep working; what it loses is the ability to answer them from cache. After the return, each read and attribute check goes to the server again, and the client’s behaviour reverts to the close-to-open model described in attribute caching and validation.

The SMB analogue, and why the two systems converged

SMB servers grant opportunistic locks — oplocks — and, from version 2.1 onward, leases with three independent bits: read, write, and handle caching. The shape of the mechanism is the same as a delegation. The server grants caching authority when it believes the file is not shared, and it must break the grant when another client’s request conflicts.

The break is a request in the other direction, carried on the same connection rather than on a separate callback channel, and the client must acknowledge it by downgrading or releasing its cached rights. The break notification is asynchronous from the server’s point of view: the server sends it and waits for the acknowledgement, with a timeout after which it proceeds as though the break had completed.

The differences are instructive. Because SMB’s channel is the session itself, a break can only be delivered while a connection is live, and a client with a broken connection cannot be informed. Because the rights are three independent bits rather than two modes, a server can downgrade partially — from read-write to read-only — instead of revoking everything. And because SMB was designed for clients that may hold thousands of handles, the break protocol is tuned for volume, whereas the NFS delegation protocol is tuned for the case where the server has a handful of delegations to recall and can afford a round trip per recall.

Both designs share the same unresolved problem: a client that has been partitioned cannot be told that its authority is gone, and the server has to choose between waiting and revoking. Every choice here trades a stalled operation against a silent violation of caching, and the setting that is best in one environment is wrong in another.

Limits of the delegation model

A delegation optimises the private case, so it cannot help a workload that shares files. Once conflicting access appears, the server stops granting delegations for that file and the filesystem falls back to per-request validation. Workloads that scale by having many readers on one file gain nothing from the mechanism. The mechanism that makes this manageable is in the open: a client that knows it will not benefit asks for no delegation, and a server that holds a conflicting open or lock simply returns no grant, so the second opener never waits on a recall it did not need. Delegations also interact awkwardly with locks, because the two are different forms of the same idea. A client with a write delegation has exclusive access by construction, so a lock request from another client forces a recall. Conversely, a client that holds a lock and no delegation still sends every operation to the server, and the delegation’s presence tells the server that no other opener exists to be invalidated.

The final limit is that delegations must be reclaimed after a server restart, along with opens and locks, which is the recovery problem described in grace periods and state reclaim. A client that reclaims a delegation it held before the restart gets its caching authority back; a client that fails to reclaim in time is left sending requests, which is slower but always correct. The mechanism degrades to the per-request model rather than failing, and that property is what makes it safe to enable by default on servers that support it.