Backend Systems · Cluster
Background Jobs
Scheduling, idempotency, and retries.
- 01 The Job Record State Machine: Enqueue, Claim, and Acknowledge How a durable job queue represents work as a record with an explicit state machine, why claiming must be a single conditional update, and which guarantees the record cannot provide.
- 02 Visibility Timeouts and Lease Heartbeats in Job Claims Why a claimed job stays hidden from other workers only until a deadline, how heartbeats extend that deadline safely, and what happens when the worker is alive but cannot renew.
- 03 Idempotency at the Effect Boundary, Not the Handler Entry Deduplicating a job at the point where it changes the world: unique constraints, request keys, delta-to-fact conversion, and the two-phase effect that survives a crash mid-call.
- 04 Retry Ladders and Dead-Letter Promotion for Failed Jobs How job retries differ from request retries: escalating delays held as queue state, caps in wall-clock time, crash loops, and what promotion to a dead-letter state must preserve.
- 05 Cron Triggers, Misfire Policy, and the Runs That Never Happen Turning a recurrence expression into at-most-one job per period, choosing what to do with slots missed during downtime, and why a schedule that never fires is invisible without an external expectation.
- 06 Holding Delayed Jobs: Buckets, Timers, and the Cost of Time-Ordered Scans Why millions of future-dated jobs concentrate contention on one index page, how time bucketing and a separate promotion step spread it, and where in-memory timer wheels fit.
- 07 Priority Classes and Tenant Fairness in Shared Job Queues Why a single ordered queue lets one tenant or one class consume the whole worker pool, how weighted dispatch and per-tenant caps differ, and what to measure when fairness is the goal.
- 08 Fencing Tokens and the Worker That Did Not Notice It Was Replaced How a monotonic lease epoch stored beside the protected data stops a stale worker from committing, why a timestamp cannot serve as the token, and what the guarantee excludes.
- 09 Chained Jobs and Fan-In Completion Without a Coordinator How a job that spawns children can know when they are all finished, why the counter must count terminal outcomes rather than successes, and how the successor is triggered exactly once.
- 10 Draining a Worker Pool Without Losing or Doubling the Work The shutdown sequence for a job worker: stop claiming, cancel at a deadline, release leases instead of waiting for expiry, and keep heartbeats alive until the handlers stop.