Coarse-Grained Locking by Design: What a Chubby Cell Optimizes For

A coordination service is not a general-purpose database, and the choices that make it good at one coordination pattern make it poor at another. The clearest way to see this is to compare the two assumptions a lock service can be built around: that locks are taken rarely and held for a long time, or that locks are taken constantly and held briefly. Chubby was designed explicitly around the first assumption, and almost every structural decision in it follows from that choice.

The workload the design assumes

Chubby’s stated purpose is coordination among loosely coupled systems: electing a leader, letting a leader advertise its location to followers, and storing small amounts of configuration that must be consistent. Locks in that setting are taken when a process starts or when leadership changes, held for hours or days, and released when the process exits. A cell may serve a handful of clients with a handful of locks each, and the lock service’s throughput requirement is modest.

That assumption changes what matters. If locks are held for hours, the cost of acquiring one is irrelevant next to the cost of holding it, and the design should minimize the per-lock overhead on the server rather than the per-acquisition latency. If locks are taken rarely, the number of clients that can be served by one cell is bounded by the number of locks and files they need, not by the rate of operations. And if the locks protect coarse operations, such as “this process is the leader for this shard”, then the granularity of the lock is coarse, and the amount of work done under it is large enough that the lock’s own latency is noise.

The paper is direct about the consequence: the design is not intended for fine-grained access to high-churn data, and using it that way would be a mistake. A key-value store with a lock per record, taken and released thousands of times a second, has the opposite workload, and the techniques that make Chubby efficient would be actively harmful there.

A cell and its master

A Chubby cell is a small set of replicas, typically five, that maintain a consistent replicated filesystem-like namespace. One replica is the master, elected through a consensus protocol among the replicas, and all client requests that mutate state go through the master. The master holds a lease on its mastership, which is what lets it serve requests without consulting the other replicas on every one: while the lease is valid, the master knows the others will not elect a different master, and it can act as the single point of serialization for the cell.

This is the structure that makes the service affordable. A consensus protocol that ran on every client request would multiply the cost of each operation by the number of replicas and by the round trips between them. Electing a master once and letting it serve from its lease turns the per-request cost into a local operation plus, for durability, a write to a log that is replicated asynchronously or in the background. The lease on mastership is the same mechanism as the client-facing lease described in the lease as a time-bounded grant, applied to the server’s own leadership.

The master’s lease creates the same safety obligation as any other lease. If the master pauses past its lease and a new master is elected, the old master must not continue to serve writes. It detects the situation by failing to renew, and it must stop serving at its own deadline rather than at the moment it learns of the new master. A master that keeps serving after its lease expires is a second writer to the replicated log, and the resulting divergence is exactly the failure the election was meant to prevent.

Caching lock state at the client

The largest cost in a lock service is the round trip per lock operation, and the largest saving is to avoid the round trip when the answer is already known. Chubby clients hold a handle to an open file or lock, and the handle caches the lock state and the file contents. A client that checks whether it holds a lock can often answer from the cached handle without contacting the master, because the handle was granted a lease and the master will not invalidate it without notifying the client.

The cache is not a local optimization with a correctness hazard; it is a lease with the same expiry semantics as everything else. The handle’s cached state is valid for the duration of its lease, and the master’s guarantee is that it will not grant a conflicting lock during that window. When the lease is close to expiry, the client must renew, and a client that fails to renew must treat its cached state as void. The reason caching is safe is precisely that the cache’s validity is time-bounded and the server enforces the same bound.

This is where the coarse-grained assumption pays again. A client that holds a lock for hours renews its handle a few times an hour, and the amortized cost of the lock is negligible. A client that holds a lock for a millisecond would spend more time renewing the handle than holding the lock, and the caching machinery would be pure overhead. The design is coherent only for the workload it targets.

Events instead of polling

Chubby notifies clients of changes through events on open handles, in the same spirit as a watch. A client that needs to know when a file changes, or when a lock becomes free, subscribes and receives an event rather than polling. The event delivery is subject to the same caveats as any notification: the event says that something changed, and the client re-reads to find out what.

The design choice that follows from coarse granularity is that events can be delivered with modest latency without stressing the master. If a cell serves a few hundred clients and each has a few open handles, the master’s event bookkeeping is small. A cell serving millions of handles would need a different structure, because the master would be maintaining per-handle state for a population far larger than the number of locks it manages.

Where the design stops paying

The failure mode of a coarse-grained lock service is a client that treats it as a fine-grained one. Every acquisition becomes a master round trip that the cache cannot amortize, every release becomes a write to the replicated log, and the master’s throughput becomes the limit for the whole cell. Since the master is a single replica serving all mutations, its capacity is the cell’s capacity, and a cell sized for a few hundred coarse locks will saturate under a workload of thousands of short ones.

The second failure mode is a lock held too long by a client that has stopped making progress. A coarse lock held for hours is fine when the holder is doing hours of work; it is a stall when the holder is blocked on something unrelated. Because the lock’s lease is long, the service will not reclaim it quickly, and the recovery time is the lease duration plus the margin. A system built on long leases must therefore be built to detect a stuck holder and to release the lock deliberately, because the lease’s expiry is too slow to be the primary recovery mechanism.

The third is the one that applies to every lock service and is sharpest here: the lock’s protection is only as good as the resource’s cooperation. A client that holds a Chubby lock and writes to a storage system that does not check a sequencer is protected only while its lease is valid and only against clients that also respect the lock. The service’s coarse granularity makes this more likely to go unnoticed, because the lock is held so long that the window in which a stale holder could write seems negligible. It is not negligible; it is the same window, measured against a longer lease.

The comparison that matters

The instructive comparison is with a service designed for the opposite workload. ZooKeeper’s design assumes many clients, many watches, and frequent small operations, and it optimizes for throughput of reads and for a large number of sessions. Its consistency model is built around a leader that serializes writes and followers that serve reads, with watches to keep clients informed. The two services solve overlapping problems with different assumptions, and the assumptions determine which is appropriate.

The lesson is not that one design is better. It is that “a lock service” is not a single specification, and the properties a client needs from one depend on how the client uses it. A service that promises linearizable locks and delivers them through a single serialization point has a capacity bound, and the bound is a function of how often clients ask. A client that asks once an hour has a service that will never saturate; a client that asks a thousand times a second has a service that will, and no amount of client-side cleverness changes the master’s throughput.

Choosing between them is a matter of measuring the workload’s shape: the number of locks, the hold time, the acquisition rate, and the tolerance for the service being unavailable. Those four numbers determine which of the designs fits, and the mistake is to pick by reputation or by the convenience of an existing client library. The mechanisms are well understood; the fit is what requires judgment.