Visibility Timeouts and Lease Heartbeats in Job Claims

A worker claims a job and starts executing it. If the worker then dies, nothing in the system observes the death directly: there is no notification, no socket close that the queue can attribute to the job, no supervisor with authority over the claim. The only recoverable design is to treat the claim as valid for a bounded period and to let the record become claimable again when that period elapses without renewal. That bound is called a lease in the distributed-systems literature and a visibility timeout in queue services, and it is the mechanism that makes a crashed worker indistinguishable from a slow one in a way the system can still act on.

A claim that expires on its own

The claim records a deadline, and the claim predicate includes it. A job is available to any worker when its status is runnable or when its lease has expired:

SELECT id FROM jobs
WHERE (status = 'pending' AND available_at <= now())
   OR (status = 'claimed' AND lease_expires_at < now())
FOR UPDATE SKIP LOCKED
LIMIT 1;

Some systems model this with a lock instead of a status change: SQS hands out a receipt handle and makes the message invisible for the duration of the visibility timeout, then returns it to the queue. The default visibility timeout there is 30 seconds and the maximum is 12 hours, with a message retained for at most 14 days before it is discarded regardless of how many times it was received. A database-backed queue normally encodes the same idea as a nullable deadline column, because that keeps the state visible to ordinary queries and lets expiry be handled by the same UPDATE that claims.

Expiry-based recovery has one property worth stating explicitly: the system never needs a failure detector. There is no heartbeat monitored by a coordinator, no agreement about who is dead, and no divergence between two observers’ conclusions. Each worker decides for itself, from the record, whether a job is claimable — and the storage engine’s row lock arbitrates when several decide at once.

The cost is latency of recovery. A worker that dies mid-execution leaves the job claimed until its deadline passes. If the lease is five minutes and the crash happens one second after claiming, the job is stalled for four minutes and fifty-nine seconds before any other worker can see it. Short leases reduce that stall; long leases are safer against duplicate execution. The number is a dial between two failure modes.

The two failure directions of one number

Set the lease shorter than the handler’s running time and the job is claimed again while the first worker is still working. Both execute. For a handler that sends mail or charges a card, that is a duplicate effect; for one that writes a row keyed by an identifier it may be a harmless overwrite or a constraint violation, depending on the schema.

Set the lease longer than necessary and every crash converts into a stall of that length. Combined with a retry policy that only starts counting after the expiry, a slow recovery lease also stretches the total time to completion for a job that needed several attempts, as the ladder in Retry Ladders and Dead-Letter Promotion assumes when it computes a wall-clock deadline.

The parameter that matters is the ratio between the lease and the handler’s execution time distribution, not the absolute value. A lease covering the p99.9 of handler duration, plus headroom, keeps duplicates rare without turning crashes into long stalls. A lease set to the maximum observed duration is fragile in a different way: the maximum is not stable, and a job that suddenly runs longer — a larger input, a slower dependency — silently crosses it.

Services that cannot tolerate the stall push the other way and use a short lease plus renewal. That is where a heartbeat comes in.

Heartbeats and the conditional extension

A heartbeat is a periodic write that moves the deadline forward while the handler continues:

// extendLease returns false when the claim was lost and the handler must stop.
func (w *Worker) extendLease(ctx context.Context, id, token string) (bool, error) {
	tag, err := w.db.Exec(ctx, `
		UPDATE jobs
		SET lease_expires_at = now() + interval '90 seconds'
		WHERE id = $1 AND claim_token = $2 AND status = 'claimed'`,
		id, token)
	if err != nil {
		return false, err
	}
	return tag.RowsAffected() == 1, nil
}

The RowsAffected() == 1 check is the whole value of the exchange. Zero rows means the lease was already reassigned, or the job was acknowledged elsewhere, or the row moved to a terminal state. A worker that keeps running after a failed extension is working on a claim it no longer holds, and whatever it writes at the end will be rejected if the acknowledgement is conditional — but the side effects it already performed will not be.

Three parameters interact: the lease length, the heartbeat period, and how far each heartbeat extends. Extending by the full lease on every beat is the common choice, which means a worker that loses a single heartbeat still has nearly a full lease left before it becomes claimable again. A period of one third to one fifth of the lease tolerates several consecutive missed beats. Shorter periods cost more writes, and each extension is a durable write unless the storage batches it.

That last point produces a real failure mode on database-backed queues. Heartbeats for many jobs contend with the claim and acknowledgement traffic on the same tables, and a burst of heartbeat writes can stall the connection pool that the handlers themselves need. Two mitigations are usual: extend by a longer interval per beat so beats are rarer, or keep a separate lock-table with a narrower row so extension does not touch the wide job row.

Clocks that are not the same clock

Every comparison above is between timestamps, and timestamps come from clocks that disagree. Two hosts synchronised by NTP typically agree to within a few milliseconds and can differ by seconds after a suspend, a virtual machine pause, or a leap-second adjustment. A machine with a badly drifting clock can be minutes off.

The workable rule is that lease arithmetic happens on one clock: the storage’s. now() in the update above is evaluated by the database server, and expiry is compared against the same server’s clock during the claim. The worker never computes its own deadline and never writes a timestamp it produced itself. Where workers must reason about the deadline locally — to decide whether to attempt a beat at all — they should compare their own monotonic clock against an interval, not against a wall-clock timestamp taken from the record.

A worker that uses its own clock to decide that its lease is still valid can be wrong in both directions. Ahead of the server, it believes it has time it does not have and keeps executing past expiry. Behind, it abandons claims that are still valid, which wastes work and produces spurious retries.

Alive but unable to renew

The interesting failure is not the crashed worker. It is the worker that is running, executing its handler, and unable to reach the queue: a network partition to the database, a saturated connection pool, a process stopped by the kernel for a live migration, a runtime whose event loop is blocked by a long synchronous call.

In each of those cases the lease expires and the job is reassigned while the original worker continues. A partition heal later, the first worker finishes and discovers its acknowledgement is rejected. The second worker may have finished the job minutes earlier. The system as a whole behaved correctly — it made forward progress — but the side effect happened twice.

This is why a heartbeat watchdog that aborts the handler is worth its complexity. The handler receives a cancellation signal the moment an extension fails, so the window between losing the claim and stopping is one heartbeat period rather than the whole remaining runtime. Handlers that respect context cancellation and check it between units of work narrow the exposure further; handlers that ignore it widen it.

Blocking heartbeats from the same thread that runs the handler defeats the mechanism entirely. If the handler occupies the only worker thread, or holds the lock the heartbeat needs, the lease expires exactly during the period of most active work. The heartbeat must run on an independent path — a separate task, a separate thread, or a timer that fires from the runtime’s scheduler.

When the handler outlives any reasonable lease

Long jobs — a backfill over a large table, a video transcode, a nightly reconciliation across many shards — cannot be covered by a lease without either making the lease enormous or accepting constant renewal for hours.

The usual structure is to break the job into units that each fit comfortably inside a lease, with the job recording a cursor between units. Each unit renews the lease, commits its progress, and releases; a crash resumes from the last committed cursor rather than from the beginning. The lease then bounds only the unit, and the recovery stall is bounded by the unit’s duration rather than the whole job’s.

The alternative is a job that extends its own claim indefinitely and is never re-claimable while the worker lives. That works until the worker’s network is cut, at which point the job is stalled for as long as the lease; making the lease long enough for the whole job converts every crash into an outage of that job’s progress.

Whichever structure is chosen, the lease guarantees at-least-once execution and nothing stronger. Every mechanism that follows — the conditional acknowledgement described in The Job Record State Machine, the deduplication at the effect boundary, the fencing token that stops a stale writer from committing — exists because a lease can expire while the work continues.