Hard and Soft Mounts: What a Client Does When the Server Stops Answering

The server’s network cable is unplugged. A process on a client calls read on a file in a mounted directory. Nothing arrives, nothing times out at the application level, and the call does not return. It sits inside the kernel waiting for a reply that will never come, and on a mount configured the old way every process that touches the same filesystem joins it.

The mount option decides what happens next, and the options are not variations on one theme. A hard mount retries forever, which makes the filesystem behave like a very slow disk. A soft mount gives up after a configured number of retransmissions and returns an error, which makes the failure visible but can corrupt data. A third setting controls whether a signal can end the wait. Each is correct for a different kind of program, and the choice cannot be made once for a whole site.

The state a waiting process is in

When a client sends a request and waits, the waiting task is put to sleep by the kernel, and what happens to that task is governed by the mount’s retry policy. A hard mount never gives up. The rationale is specific: the operation may already have been executed on the server, and returning an error would tell the application that a write failed when it in fact succeeded. The protocol’s retransmission layer keeps resending, so the operation completes once the server returns, and the application’s view stays consistent with what actually happened.

The cost is that the process is stuck for as long as the server is unreachable. On older clients the sleep was uninterruptible, so kill -9 did nothing: the signal was delivered and noted, but the task was not in a state where it could act on it. A ps listing showed the process in the D state and it stayed there, and the only remedies were to restore the server or reboot the client. That behaviour gives hard mounts their reputation, and it is the reason a single failed server could fill a machine’s process table with unkillable tasks.

Current Linux clients keep the retry policy but make the wait interruptible, so a signal wakes the task and the call returns EINTR instead of hanging. The application then chooses whether to retry. The risk is the same one a soft mount carries: aborting an operation whose outcome is unknown leaves the client’s idea of the file out of step with the server’s, and the application that retries later may duplicate an effect the first attempt already had.

What a soft mount actually promises

A soft mount turns the wait into a bounded retry sequence. The client sends the request, waits for the RPC timeout, retries, and after the configured number of retransmissions fails the request and returns an error to the application. Which error depends on the option: soft returns EIO, and softerr returns ETIMEDOUT. The distinction exists because the two errors mean different things to a caller — EIO reads as a device fault, ETIMEDOUT as a timeout — and neither distinguishes a read from a write, because the client does not know which operations the server received.

The problem is that the error is a lie about the operation’s outcome. The client gave up; the server may still be working. A write that the client has abandoned may land on the server a moment later, and the application has already been told it failed and may have taken a compensating action — retrying the write, or reporting failure to a user, or aborting a transaction. The data on the server then reflects a write the application believes did not happen.

There is a worse version of the same problem. A client that gives up on a write may have already sent the data, and the server’s reply may be in flight when the client abandons. The reply arrives, finds no matching outstanding request, and is discarded. The server has the data; the client believes the write failed. Nothing in the protocol corrects this, because the client has moved on.

For reads the consequences are milder, since a read that times out and is abandoned leaves nothing behind. That is why soft mounts are sometimes recommended for read-only mounts and discouraged for read-write ones. The recommendation is about the operation’s side effects, not about the transport.

The parameters interact in a way that is easy to get wrong. A short wait with many retries recovers quickly from a transient loss and also gives up quickly on a real outage; a long wait with few retries tolerates a longer outage but takes longer to notice a dead server. Because the per-attempt wait grows with each retransmission, the worst-case time before an error is returned is the sum of the successive waits, not the first wait multiplied by the count. The values are also capped by the client: a timeo larger than the transport’s default is reduced to the default rather than honoured.

/* Shape of the client's retry loop, simplified. */
for (attempt = 0; attempt < retrans; attempt++) {
    send_call(req, xid);
    if (wait_for_reply(xid, timeout) == OK)
        return reply;

    timeout = min(timeout + timeo, max_timeout);  /* TCP: linear, capped */
    if (soft && attempt == retrans - 1)
        return -EIO;                              /* outcome unknown */
}
/* hard mount: retry again, indefinitely */

The backoff exists because a server that is overloaded needs more time, not more requests; retrying at a fixed interval against a struggling server adds load precisely when the server can least afford it. The shape of the backoff is transport-dependent. Over TCP the Linux client starts from a 60-second wait and increases it by the configured timeo after each retransmission, up to a maximum of 600 seconds. Over UDP it adapts a timeout per request type and doubles it after each retransmission, up to a maximum of 60 seconds, falling back to the timeo value for request types it does not see often enough to measure. The retry budget is the retrans option: three attempts for a UDP request and two for a TCP one if it is not set.

The middle ground, and why it is not a compromise

A third setting controls whether the wait can be interrupted by a signal, and it is a different axis from hard and soft. Hard and soft decide what happens when the retry budget is exhausted. Interruption decides whether a signal can end the current wait.

The two dimensions used to be independent. A mount could be hard and interruptible: it retried forever, but a signal aborted the current wait, so an operator could clear a hung process without rebooting while the filesystem’s durability contract stayed intact for everything that was not interrupted. FreeBSD still exposes this as intr, and it is the setting that avoids both the unkillable-process problem and the false-error problem.

On Linux the distinction has collapsed. nfs(5) documents intr as an option kept for backward compatibility and ignored since kernel 2.6.25, because hard-mount waits are interruptible by default. A signal delivered to a task waiting on a hard mount makes the call return EINTR, and the application decides whether to retry. The mount option is still accepted so that existing mount tables parse, which is why its documentation and its behaviour no longer agree.

The value of interruptibility does not depend on the interface. It makes the wait respond to signals without making the operation fail. The application sees EINTR and can decide whether to retry, which is the contract POSIX specifies for interruptible system calls. The application that retries gets the operation completed once the server returns; the application that does not retry has chosen to abandon it, and the consequences of that choice are the application’s responsibility rather than an artefact of the mount option.

What the mount option cannot fix

All three options are local policies applied to a remote failure, and none of them can make the failure benign. The underlying problem is that the client cannot distinguish a server that is down from a server that is slow, and no local policy resolves that ambiguity.

The most dangerous case is a server that is alive but partitioned from one client. A hard mount waits, correctly, because the server will come back. A soft mount gives up, and if the application was writing, the data may have reached the server anyway. Neither behaviour is wrong in the abstract; the wrongness depends on whether the application can tolerate an unknown outcome.

A second case is a server that has restarted. The client’s requests arrive at a server that no longer recognises the file handles it issued, and the replies are ESTALE rather than silence. A hard mount does not wait here — the error is immediate and definitive, and the operation fails. This is the case where the mount option is irrelevant, because the failure is in the protocol rather than in the transport.

A third case is a server that answers but is slow because of its own load. The client’s retries add more requests to an already loaded server, and the growing wait between attempts helps only after several failures. There is no mount option that makes the client stop asking; that would require the server to signal load, which the protocol does not provide.

The fourth case is the one that produces the worst outages. A client with a hard mount on a directory that its boot process needs — a home directory, a binary path, a library path — will hang during boot if the server is unavailable. The boot process waits on a filesystem that cannot answer, and nothing proceeds. This is why system startup ordering matters on clients that mount remote filesystems, and why the mount options for such filesystems are usually chosen to fail rather than wait. The alternative, a soft mount, lets the boot proceed with errors, which is preferable to a machine that never finishes starting.

Choosing by what the application can tolerate

The decision reduces to a question about the application’s tolerance for an unknown outcome. An application that can detect a failed write and redo it, with a way to determine whether the redo is safe, can use a soft mount. An application that cannot distinguish a lost write from a completed one should not, because the error it receives is not actionable.

For reads, the analysis is simpler. A read has no side effect, so abandoning it is safe; the application retries or reports an error, and the filesystem is unchanged. This is the basis for treating read-mostly mounts as candidates for soft behaviour, and for treating write-heavy mounts as candidates for hard.

The default on both Linux and FreeBSD is a hard mount, because it preserves the durability contract. The pathological case — an outage that hangs every process touching the mount — is accepted as the cost of never lying about whether a write landed. Linux additionally makes the wait interruptible, so the cost is a stalled call rather than a stalled call that cannot be signalled; FreeBSD requires intr for that. Clients whose boot depends on a remote filesystem use a different mechanism to avoid the hang: FreeBSD’s bg and retrycnt let the mount attempt continue in the background instead of blocking startup, and the same effect is reached on Linux by giving the mount unit a startup timeout so that boot proceeds without it.

One further mechanism changes the shape of the problem rather than the policy: a client that holds enough cached authority to answer operations during an outage. That is what a write delegation provides for a single file, as described in delegations and callback channels, and a client that keeps answering from cache after the server has gone silent is choosing to extend that authority past the point where the server could recall it. The choice turns an outage into a delay at the cost of a window in which two clients can both believe they hold the same file. No mount option can make that trade safe; only a protocol that records and replays the changes can, and the recovery machinery it needs is the same machinery that governs state reclaim after a server restart.