One-Shot Watches and the Changes That Fall Between Them

A client that needs to know when a value changes has two options: poll, which wastes requests when nothing changes, or subscribe, which wastes nothing when nothing changes but creates a delivery problem when something does. Subscription is the obvious choice for coordination state, because coordination state changes rarely and the latency of a change matters. The delivery problem is that a notification is not the change; it is a message about a change, and the two can be separated by an arbitrary interval in which the state moves again.

The register-and-fire cycle

The common design is a one-shot watch. A client registers interest in a path, the service records the registration, and when the path next changes the service fires the registration and discards it. The client receives one notification, and if it wants another it must register again. The reason for one-shot rather than a persistent subscription is that the client is expected to re-read the state after every notification anyway, so the notification only has to mean “look now”, and a registration that survives the notification would either pile up duplicate deliveries or require the service to track per-client state across re-reads.

The cycle has a gap in it. Between the notification arriving and the new registration being processed by the service, a change can occur that fires no registration, because the old one has already been consumed and the new one does not exist yet. If the client registered after the change, it will be notified of the next change, not the missed one, and it will believe it is up to date while holding a stale value. If the client registered before the change but the notification crossed it in flight, the client may process the notification and then re-register, and see no further notification until the next change, which is correct only if the re-read that accompanied the notification saw the missed change.

The fix is not to close the gap but to make the client’s protocol tolerant of it. The client registers the watch, then reads the state, then processes whatever the read returned. If a change happened between the registration and the read, the notification for it may arrive after the read, and the client will re-read again and see the newer state. If the change happened after the read and before the next registration, the client’s read is stale but the notification from the old registration fires, prompting another read that sees the change. The ordering that makes this work is registration before read, not read before registration, because a registration made after a read can miss a change that occurred between them.

// Register first, then read. A change in between fires the registration and
// forces another iteration; a change after the read is caught by the watch.
for {
	ch, err := watch(path)
	if err != nil {
		return err
	}
	state, ver, err := readWithVersion(path)
	if err != nil {
		return err
	}
	if apply(state) {
		return nil
	}
	select {
	case <-ch: // "something changed": loop and re-read
	case <-ctx.Done():
		return ctx.Err()
	}
	_ = ver // the version, not the event payload, is the durable fact
}

What the notification carries, and what it does not

A notification that says only “this path changed” is safer than one that carries the new value, because a value delivered with a notification is already stale by the time it is processed. The client has no way to know whether the value it received is the latest, and a client that acts on it directly can act on a state that a later notification has already superseded. A notification that carries the new value and a version is better, since the client can compare versions and discard the older one, but the client still needs a read to be sure it has the newest state, because the notification’s version may be behind a change that has not yet produced a notification.

Services that deliver events with payloads usually pair them with a revision number on every event and a rule that the client must not assume events are contiguous. A client that receives revision 12 and then revision 15 knows it missed 13 and 14 and must re-read rather than apply 15. A client that receives 12 and then 13 can apply both in order if the events are guaranteed to be delivered in order per path, which many services do guarantee per connection but not across reconnects.

The reconnect is where the guarantee is weakest. A client whose connection drops loses its registrations, and the service may not have retained the events that occurred while it was away. On reconnect, the client typically re-registers and re-reads, which is correct but only if the re-read happens before the client acts on any cached state. A client that resumes from its cache and re-registers in the background can spend an arbitrary interval acting on state that changed during the outage.

Coalescing, and why fewer notifications is not fewer changes

Under a path that changes frequently, delivering one notification per change is wasteful. Services therefore coalesce: a client registered on a path that changes ten times receives one notification, or one per connection flush, rather than ten. The client cannot tell the difference between one change and ten, and this is a feature. The notification’s meaning is “the state is not what the client last read”, and one such message is sufficient regardless of how many changes produced it.

The coalescing has a cost in latency and a benefit in load. A client that needs to know about every individual change, rather than the aggregate, cannot use a coalescing watch; it must use a log or a sequence of events with explicit ordering. That is a different mechanism with different guarantees, and the two should not be confused. A watch is a hint; an event log is a record.

The load argument matters at scale. A directory with ten thousand children and a thousand watchers generates a notification fan-out on every child creation, and the service’s cost is the product of the two. Services bound this by limiting watches per connection, by firing a parent’s watch only for child additions rather than for every descendant change, and by refusing to hold watches on very large directories. A client that needs to monitor a large population should watch the parent for membership changes and read the children in bulk, rather than watch each child individually, because the latter multiplies the service’s work by the size of the population.

Ordering between watches and writes

A watch fires when a change is applied, but the notification travels on the client’s connection and competes with the client’s own requests. A client that issues a write and then waits for a notification of its own write can deadlock if the service does not fire watches for the client’s own changes, or can see the notification before the write’s response if the service fires them in a different order. Most services guarantee that a client’s own operations are answered before any watch event caused by them, on the same connection, so a client that writes and then reads sees its own write and then the event. The guarantee is per connection and does not survive a reconnect.

The more useful rule is that a watch is not a synchronization primitive and should not be used as one. Waiting for a notification is waiting for a message, and a message can be lost, delayed, or coalesced. The synchronization primitive is the read that follows, and the read is linearizable, so it returns the state at a definite instant. A client that structures its protocol as “read, decide, conditionally write, re-read on failure” does not need watches at all; watches are an optimization that replaces polling with a hint, and the protocol must be correct with the hints removed. This is the same discipline as conditional writes and version checks, applied to the read side.

Firing on session expiry

An ephemeral node’s deletion is a change like any other, and watchers on the parent directory are notified. The notification does not say whether the node was deleted deliberately or because its session expired, and the client’s reaction should usually be the same: re-read the directory and recompute. The distinction matters only for protocols that need to know whether a member left cleanly, which is rare, and for those the member’s own farewell message, written before it closes its session, is more reliable than inferring intent from the deletion.

A watch on a node that is deleted fires once and is then gone, because the node no longer exists. A client that wants to know about recreation must watch the parent, not the node, since the recreated node is a different node with a different version even if it has the same name. This is a common source of confusion in membership protocols: the member’s node is deleted and recreated on rejoin, and a watcher attached to the old node never hears about the new one. Watching the directory is the dependable choice, at the cost of receiving notifications for changes to members the client does not care about.