The Job Record State Machine: Enqueue, Claim, and Acknowledge

Work that a request handler cannot afford to do inline — sending a confirmation mail, rendering a report, reconciling a ledger against an upstream feed — is handed to another process through a durable record. That record is the whole contract between the two sides. It outlives the process that created it and the process that executes it, so everything the executor needs must be stored in it, and its state must be reconstructible from storage alone after any crash. A queue held in process memory, or in a channel between goroutines, satisfies none of that: the first restart discards the backlog and no caller ever learns which units of work were lost.

The durable form of the same idea is a row with a status column, or a message with a receipt handle. Both are the same design wearing different vocabulary, and both reduce to a small set of transitions that have to be atomic with respect to other workers.

One record, five states

A job passes through a bounded set of states. The names vary between systems; the shape does not.

• Enqueued but not runnable yet: the row exists, with an available_at timestamp in the future. Delayed jobs, cron-generated jobs, and retry backoff all use this state rather than a separate mechanism.
• Runnable: available_at has passed and no worker holds the job.
• Claimed: one worker has taken responsibility until a deadline, recorded as lease_expires_at.
• Terminal success: the handler completed and the result was recorded.
• Terminal failure: the retry policy is exhausted, or the failure was classified as permanent.

Two invariants make the rest of the design work. A job has at most one claimant at a time, and terminal states are absorbing — nothing moves a succeeded job back to runnable, so a repeated acknowledgement is a no-op rather than a second execution.

The hot row should stay small. An attempt counter, the last error string, and a claim identity are worth keeping inline; a full history of every attempt is not, because every claim and every failure would rewrite the same row and every read of a runnable job would drag the history along. Attempt records belong in a table keyed by job identifier and attempt number, written once and never updated.

Claiming as one conditional update

The naive claim is a read followed by a write: select a runnable row, then update it. Two workers doing this concurrently select the same row and both update it, and both execute the job. Adding a check in application code between the two statements narrows the window without closing it, because the check and the write are not the same operation.

The correct claim is a single statement whose WHERE clause carries the state predicate. The storage engine’s row lock serialises the two claimants, and the loser’s UPDATE matches zero rows.

UPDATE jobs
SET status = 'claimed',
    claimed_by = $1,
    claim_token = gen_random_uuid(),
    lease_expires_at = now() + interval '5 minutes',
    attempt = attempt + 1
WHERE id = (
  SELECT id FROM jobs
  WHERE status = 'pending' AND available_at <= now()
  ORDER BY available_at
  FOR UPDATE SKIP LOCKED
  LIMIT 1
)
RETURNING *;

SKIP LOCKED is the clause that turns this into a queue. Without it, a second worker blocks behind the first worker’s row lock, so every claim serialises on the head of the queue and the number of workers stops mattering. With it, locked rows are passed over and the scan continues to the next runnable row. PostgreSQL has supported the clause since 9.5 and MySQL since 8.0.

The scan itself is worth a thought. Without a partial index such as CREATE INDEX ON jobs (available_at) WHERE status = 'pending', every claim walks rows that can never be claimed, and the cost grows with the dead backlog rather than with the runnable set. A partial index keeps the claim proportional to the number of jobs actually ready to run, which is the only quantity that stays bounded under load.

Each claim is a write transaction and therefore costs a durable commit, one flush of the write-ahead log. That sets the throughput ceiling of a database-backed queue at roughly the storage’s commit rate per second, divided by the number of state transitions per job — typically two, one at claim and one at acknowledgement. Claiming a batch of rows in one statement, and acknowledging a batch in one statement, amortises the commit across many jobs, which is why workers usually take several jobs per round trip rather than one.

The dual-write problem at enqueue

The producer of a job is usually in the middle of something else. A request handler updates an order and needs a job that mails a receipt. Two writes are involved, to two places, and there is no transaction spanning them unless the job table lives in the same database as the order table.

Both orderings are broken in a specific way. Writing the job first and committing the business transaction afterwards lets a worker claim the job while the order change is still uncommitted; the worker reads the old order, produces the wrong output, and the job is recorded as done. Committing first and writing the job afterwards loses the job entirely if the process dies in between, and no retry can repair it because the caller believes its work is finished.

The transactional outbox removes the gap: the job is inserted into a table in the same database, inside the same transaction as the business change. Either both rows commit or neither does. A separate process reads the outbox table and forwards the entries, marking them as sent. The forwarding step cannot be atomic with the marking, so a crash between the two produces a duplicate delivery. That is acceptable only if the consumer tolerates it, which is the subject of Idempotency at the Effect Boundary.

When the queue is a table in the application’s own database, the outbox and the queue collapse into the same table and the producer’s insert is already atomic with the business change. When the queue is a remote broker, the outbox is the only way to get the same property.

Acknowledging, and separating failure from silence

Acknowledgement is conditional on the claim identity, not just the job identifier:

UPDATE jobs SET status = 'succeeded', finished_at = now()
WHERE id = $1 AND claim_token = $2 AND status = 'claimed';

The token matters because leases expire. A worker that was suspended — a long garbage collection pause, a machine that was migrated, a debugger left attached — resumes after its lease has been reassigned. Without the token, its late acknowledgement overwrites a newer claim’s state and the job appears finished when the second worker is still running it. Fencing Tokens and the Stale Claim Problem develops the mechanism further; here it is enough that a conditional update turns the stale writer into a no-op.

A handler that raises an exception produces a different signal from a worker that stops existing. The first can record an error and a next attempt time. The second records nothing at all; the only evidence is that the lease expires without renewal. That asymmetry is the reason leases exist in the first place: from the outside, a crashed worker and a slow worker are indistinguishable, so the claim must have a deadline and the system must assume the worst at the deadline.

Incrementing the attempt counter at claim rather than at failure is a deliberate choice. A job whose handler crashes the process — an out-of-memory kill, a segfault in a native library — never reaches its own failure path, so a counter incremented at failure would never advance and the job would be retried forever, taking a worker down each time. Incrementing at claim means a crash loop consumes attempts and the job eventually escalates to a terminal state. The cost is that a job killed mid-flight during a routine deploy loses an attempt, which is why retry budgets for infrastructure restarts are usually generous.

What the record cannot express

The state machine gives mutual exclusion within a lease and a durable record of outcome. It does not give three other things that are often assumed of it.

Ordering is not implied. Several workers claiming several rows proceed concurrently, so a queue with a single logical partition has no defined execution order, only a defined claim order. Where order is required, it has to be encoded in the claim — the oldest runnable row per key, with the key locked for the duration, or a separate queue per key with a single consumer.

Atomicity with the handler’s side effects is not implied either. If the effect is a write to the same database, both can be committed together; if it is an HTTP call to another service, no state column makes the pair atomic, and the design has to tolerate a repeated execution.

Fairness is not implied. One table serving every tenant means a tenant with a hundred thousand queued jobs delays everyone else, because the claim order is by availability and nothing stops one tenant from filling the runnable set. Priority Classes and Tenant Fairness in Shared Job Queues covers the partitioning that fixes it.

What remains is one durable write per transition, and a claim protocol that any storage engine with row locks can implement. Every other mechanism in a job system is an elaboration on that base: leases choose how long a claim may last, retries choose what a failure means, and the drain protocol decides what happens to claims that outlive their worker.