Fencing Tokens and the Worker That Did Not Notice It Was Replaced

A worker claims a job, begins work, and is suspended — a long garbage-collection pause, a hypervisor migrating the machine, a debugger, a SIGSTOP from an operator. The lease expires. Another worker claims the job and finishes it. The first worker wakes, completes its handler, and writes its results.

Conditional acknowledgement, as described in The Job Record State Machine, rejects that late write to the job record. The job shows one completion and the queue is consistent. Everything the handler wrote to other resources along the way is still there. If the handler updated an inventory count, appended to a report, or replaced an object in a bucket, the stale execution’s writes are interleaved with the current one’s, and the queue’s own bookkeeping has no way to notice.

The record is checked; the world is not

The gap is structural. The job record’s update carries the claim identity, so the store can decide whether the writer still holds the claim. Every other write the handler performs either has no such identity or is not checked against it. Two workers running the same handler do not merely duplicate an idempotent effect; they interleave a read-modify-write sequence, and the interleaving can produce a state that neither execution alone would have produced.

A handler that reads a counter, computes a new value, and writes it back is the clearest case. Worker A reads 10, worker B reads 10 after taking over, B writes 11, A wakes and writes 11 — a lost update that no unique constraint catches, because both writes are legitimate assignments. A handler that appends a line to a report produces a duplicate that looks like a legitimate second entry.

Locking longer does not close the gap, because the lock holder can always be paused for longer than the lock’s lifetime. The only reliable construction is to make the protected resource itself reject writes from a holder that has been superseded. That requires the writer to carry evidence of its position in the sequence, and the resource to compare.

A counter from the arbiter, never a clock

The evidence is a fencing token: a value that increases with every new claim of ownership, and that every write to the protected resource carries. The resource stores the highest token it has seen and rejects anything lower.

A timestamp is not a token. Two workers reading clocks that disagree by even a few milliseconds can produce out-of-order tokens, and a clock that steps backwards produces a token lower than one already accepted, permanently blocking the newer holder. NTP-synchronised clocks agree closely, and “closely” is not “monotonically”. Lamport’s ordering argument applies directly: the only reliable order is one established by the participants themselves through communication, not one read from a local oscillator.

The arbiter is whichever component serialises the claims. When jobs are claimed by a conditional update in a database, that database is the arbiter, and a per-job counter incremented at claim is the token:

UPDATE jobs
SET status = 'claimed',
    lease_epoch = lease_epoch + 1,
    lease_expires_at = now() + interval '5 minutes'
WHERE id = $1
  AND ((status = 'pending' AND available_at <= now())
       OR (status = 'claimed' AND lease_expires_at < now()))
RETURNING lease_epoch;

Every claim therefore produces a distinct, strictly increasing epoch for that job, and the row lock serialises concurrent claims so two workers cannot receive the same value. The pair of job identifier and epoch identifies an execution unambiguously, which the claim’s random token does not: a random value establishes identity but not order, and ordering is what the resource needs.

Where the claim is issued by a queue service that exposes only an opaque receipt handle and a receive count, the token has to be manufactured. A common approach is a claim table in the application database that records (job_id, epoch) with the same conditional-update discipline, making the application’s storage the arbiter even though the queue is remote. The epoch then belongs to the claim record, and the receipt handle is only how the message is acknowledged.

The epoch rides on the write

The resource side needs no extra round trip. The comparison belongs in the predicate of the write that was going to happen anyway:

UPDATE inventory
SET quantity = quantity - $2,
    writer_epoch = $3
WHERE sku = $1 AND writer_epoch < $3;

Zero rows affected means the write was fenced: a newer execution has already written this row, and this one is a remnant. The handler treats that as a signal to stop, not as an error to retry, because retrying with the same epoch will be fenced again.

Storing the epoch on the row being protected, rather than in a separate table of high-water marks, keeps the comparison atomic with the data it protects and makes the invariant local: no row ever moves backwards in epoch. It costs one column per protected table, which is small next to the alternative of an external lock manager.

The predicate’s strictness matters. writer_epoch < $3 admits the first write of epoch 3 and rejects every later write of epoch 3 as well, including a legitimate second write from the same execution. That is usually the correct semantics — one epoch, one write, enforced — but handlers that update a row repeatedly within one execution need either a sub-counter or a <= predicate. <= stays safe against the stale worker, because two executions never share an epoch; what it admits is a second write from the execution that already wrote the row, which is the effect-level idempotency problem from Idempotency at the Effect Boundary, not a fencing problem.

Bulk operations fit the same shape with a floor instead of a per-row value: a delete or a batch update carrying AND writer_epoch < $3 evaluates the predicate per row, so it can fence some rows and leave others untouched, and the handler has to compare the affected count against the expected count to learn which happened.

What the token does and does not promise

Fencing guarantees at most one writer per protected resource at a time, in the order of claims. It does not guarantee at most one effect, and the difference is worth being precise about.

A stale execution that began before the takeover may have written successfully while its claim was still valid. Those writes are not violations; they happened when that execution was the legitimate holder. Fencing cannot undo them, because the resource accepted them at the time. What fencing prevents is the stale execution’s writes after a newer claim has written — a narrower property, and the one that keeps the resource in a state some single execution would have produced.

Nor does fencing protect anything the arbiter does not mediate. An outbound HTTP call, a message published to a broker, a file uploaded to an object store that has no conditional-write facility — none of these can compare an epoch. A stale worker’s mail is sent; the token cannot recall it. Only a receiver that accepts a deduplication key, or a write that carries a conditional precondition the store enforces, extends the guarantee past the database boundary.

There is also a liveness cost. A handler that ignores a fenced write and keeps going burns its remaining runtime and may perform further unrecoverable effects; the fenced result should abort the handler immediately, through the same cancellation path a failed lease renewal uses. And a resource whose stored epoch is accidentally set high — by a test, a restore from backup, a manual correction — will fence all subsequent writes until the claim counter passes that value again. Reset procedures have to consider both counters together.

Every write path, or the guarantee is void

A fencing scheme fails silently the moment one code path writes the protected data without the predicate. The failure is not loud: the fenced writes are rejected, the unfenced path succeeds, and the resource ends up with a value from an execution that was superseded. A single UPDATE in a maintenance script, a migration that copies rows, an administrative tool that repairs a record, each can produce a state the fencing invariant declares impossible.

Two defences are practical. The predicate can be encapsulated in a repository function that is the only way the table is written, which makes bypass visible in review rather than in production. And the stored epoch can be treated as an assertion that the current holder can verify: a handler that reads a row’s writer_epoch and finds a value greater than its own knows it has been superseded even before attempting a write, which lets it stop early rather than after the damage.

The same argument applies to the claim side. If one job type is claimed by the conditional update that increments the epoch and another is claimed by a direct status write that does not, the second type has no token and its writes look like epoch zero to every comparison. Mixed claim paths in one table are a sign the epoch belongs to the claim protocol rather than to the handler, and should be enforced by making the claim statement itself the only way to obtain work.

When there is no arbiter to ask

Some deployments have no transactional store on the path: jobs arrive from a managed queue and write directly to an external service. Fencing requires an authority whose ordering the protected resource trusts, and there is none.

In that shape, the options are to introduce one, or to accept that the guarantee is not available. Introducing one usually means a small table in the application’s database holding claim epochs, which is a durable write per claim and per write — affordable when the job rate is moderate, and the reason high-rate job streams prefer to make the effect idempotent and the queue’s at-least-once delivery harmless instead.

The two mechanisms are complementary rather than alternatives. Idempotency makes a repeated effect harmless; fencing makes a concurrent write impossible. A job system that writes to a transactional store needs fencing for the interleaving case that no key can detect, and needs idempotency keys for the duplicate case that fencing cannot see. Neither is a substitute for the other: one orders writers, the other absorbs repeats, and a job that triggers its successor from inside a transaction needs both before the chain is safe.