Sessions and Ephemeral Nodes: Tying State to a Live Connection

Group membership has an awkward shape. A process joins a set, and the set must reflect that the process is alive. If the process exits cleanly, it can announce its departure. If it is killed, it cannot. The announcement mechanism therefore has to be a liveness check rather than a message, and the check has to produce an observable effect on shared state so that other processes can see the result without polling the process itself.

The usual answer is to make membership a property of a session, and to make the session’s state disappear when the session ends. A node created inside a session is deleted when that session expires, so the membership list is simply the set of nodes currently present. Nothing has to garbage-collect entries for dead processes, because the entries are not entries for processes; they are entries for live sessions.

What a session is

A session is a server-side record with a client-chosen or server-assigned identifier, an expiry deadline, and whatever state the client has attached to it. The client keeps the session alive by renewing: a lightweight request that resets the deadline. The server expires the session when the deadline passes without renewal, at which point it removes everything the session owns in one atomic step.

The renewal interval is a fraction of the timeout, chosen so that several renewals can be lost before expiry. A client that misses one renewal because of a scheduling hiccup or a lost packet still has a live session; a client that misses enough renewals to cross the deadline loses the session and everything attached to it. The asymmetry is intentional: false expiry is expensive for the client, since it loses its nodes and must rebuild them, while late expiry is merely a delay in cleaning up after a dead client.

Session identifiers must be hard to guess, because possession of the identifier is the credential. A client that learns another’s session identifier can renew it and keep a dead process’s nodes alive, which is enough to corrupt a membership list. In practice the identifier is a large random value or a server-generated value derived from a secret, and the connection that carries a renewal is bound to the session so that a renewal from an unrelated connection is rejected.

// The client's side of a session: renew well before the deadline, and treat
// the deadline as hard. A renewal that fails is not proof of expiry; only the
// server's answer is.
type Session struct {
	id       int64
	deadline time.Time
}

func (s *Session) Alive(now time.Time) bool { return now.Before(s.deadline) }

// renewUntil blocks, sending a keepalive at a third of the remaining time, and
// returns when the local deadline passes or the caller cancels.
func (s *Session) renewUntil(ctx context.Context, keepalive func(context.Context, int64) (time.Time, error)) error {
	for {
		wait := time.Until(s.deadline) / 3
		if wait <= 0 {
			return errSessionLost // the deadline is hard, whatever the server thinks
		}
		t := time.NewTimer(wait)
		select {
		case <-ctx.Done():
			t.Stop()
			return ctx.Err()
		case <-t.C:
			next, err := keepalive(ctx, s.id)
			if err != nil {
				if !s.Alive(time.Now()) {
					return errSessionLost
				}
				continue // transient: the next iteration retries before the deadline
			}
			s.deadline = next
		}
	}
}

The ambiguity that the client must resolve

The hard part is not the renewal loop. It is what the client does when a renewal times out. A timeout means the client does not know whether the server processed the renewal, whether the server is unreachable, or whether the session has already expired and its nodes been deleted. The three cases require different responses, and the client cannot distinguish them from the failure alone.

If the session survived, the client still owns its nodes and must not recreate them; recreating a node that already exists fails, and more importantly a recreated node may carry a different sequence number or be placed at a different point in the ordering, which invalidates any decision the client made earlier on the basis of the old node. If the session expired, the client owns nothing and must rebuild its entire attached state, including re-entering any queue or election it had joined. A client that assumes the optimistic case keeps a membership entry that others have already seen disappear and reappear, and a client that assumes the pessimistic case tears down and rebuilds state that was never lost, which under a flapping network turns into churn.

The resolution is to make the question answerable rather than to guess. A client that reconnects with the same session identifier learns from the server whether the session is still valid. If it is, the client resumes. If it is not, the client treats every node it believed it owned as gone and rebuilds. The server can answer this because it is the authority on expiry; the client’s own clock is not. This is the same asymmetry as in a lease: the granter decides when a grant ends, and the holder’s belief is provisional.

There is a further wrinkle when the session is valid but the connection that carried the original state is gone. Some services require that a session’s attached state be re-established on the new connection, while others keep it server-side and only require the identifier. The difference shows up in the failure window: if the client’s state lives only on the old connection, a session that survives a reconnect may have lost its nodes anyway, and the client must reconcile. Clients that store their own view of what they created, and verify it against the server after reconnect, avoid both the churn and the silent divergence.

Ephemeral nodes as membership entries

Given a session mechanism, group membership is a directory whose children are ephemeral nodes, one per member, created by the member’s session. Listing the children gives the current membership. Watching the directory gives change notifications, though a client must re-read after each one rather than trust the event. The member’s departure, clean or otherwise, deletes its node, and the deletion is a change that watchers see.

The design has a property that a heartbeat-plus-heartbeat-timeout design does not: the failure detection is centralized and consistent. Each client checks one thing, its own session. The server checks each session against its own clock. No member has to poll every other member, and no member has to decide whether another member’s silence means death or a slow network. The cost is that the service is now the arbiter of liveness for the whole group, so a service outage causes a mass expiry and a mass rejoin when it recovers. A group of a thousand members whose sessions all expire together will produce a thousand node creations and a thousand watch notifications in a burst, and any protocol that reacts to membership change by doing expensive work will do that work a thousand times.

The mitigation is to treat membership as a set that changes rarely and to debounce the reaction. A member that sees its own session expire should not immediately recreate its node and rejoin the election; it should first determine whether the service is reachable and whether its previous incarnation is truly gone. Otherwise a service restart produces a wave of duplicate joins, and a leader election that was stable for hours re-runs because every candidate’s node vanished and reappeared in a different order.

Ephemeral state and the ordering guarantee

An ephemeral node created under a sequential path receives a sequence number from the service, and the sequence numbers of a session’s nodes are assigned in the order the creations were linearized. When a session expires and its nodes are deleted, the numbers are not reused. A client that rejoins after expiry receives new, higher numbers, which places it behind every existing member in any queue ordered by those numbers. That is usually the correct behaviour for a fair lock queue: a client that lost its session lost its place, and it should not be able to reclaim a position ahead of members that stayed alive.

The same property makes ephemeral nodes a poor fit for state that must survive a client’s failure. A configuration value, a schema version, or a partition assignment written as an ephemeral node disappears when the writer’s session ends, which is rarely what the operator intended. Persistent nodes exist for that case, and the distinction between the two is the distinction between “this record describes a live participant” and “this record describes a fact”. Mixing them up produces a system whose configuration vanishes whenever the process that wrote it restarts, and the symptom looks like a coordination failure when it is really a choice of node type.

Watches, sessions, and the cost of both

Every ephemeral node is a piece of state the server must hold and a watch target that clients can attach to. A service with a large number of short-lived sessions carries the cost of creating and deleting nodes continuously, and the deletion of a node with many watches is a fan-out event. Services that host large ephemeral populations therefore bound the number of watches per connection, batch the notifications, and sometimes coalesce them so that a client that missed several changes to the same path receives one notification. The coalescing is a deliberate weakening: the client is told that something changed, not what changed, and must re-read to find out. That is the subject of one-shot watches and missed events, and it interacts directly with session expiry, because the reconnect that follows an expiry re-establishes watches and may have missed changes in between.