Durable Deferred Work and the Persistence Boundary

An in-memory queue is a scheduling decision with no record. It exists in a process that the system may reclaim without notice, and the moment the process dies the decision dies with it: nothing remains to say that the work was pending, so nothing can restart it. A durable queue replaces the decision with a row, and the row is what makes deferred work deferrable at all.

The persistence boundary is the line between work that is merely queued and work that is guaranteed. Where that line falls determines what happens after a kill, after a reboot, and after the app is reinstalled.

A queued item is a row, not a closure

The obvious design for deferred work is to store a closure and run it later. The design fails at the first process death for a reason that has nothing to do with closures being fragile: a closure has no stable identity outside the process that created it. It cannot be reconstructed from bytes, so it cannot be written down, so it cannot outlive the heap.

What can be written down is a description of the work. The description needs four parts:

Field	Purpose	Failure if absent
Stable identifier	Deduplication across retries and duplicate enqueues	The same work runs twice after a crash between execution and acknowledgement
Kind plus parameters	Enough to reconstruct the operation in a fresh process	Nothing to run after restart
State plus attempt count	Resume point and retry accounting	Infinite retry, or a permanently stuck row
Constraints	Network, charging, idle, battery	Work runs in a state where it cannot succeed

The row is committed to a local database before the caller is told that the work is scheduled. That ordering is the whole guarantee: a commit that happens after the acknowledgement leaves a window in which the caller believes the work is durable and the disk has no record of it. Inverting the two is the single most common defect in hand-rolled deferred-work systems, and it is invisible until the process is killed inside that window.

SQLite is the usual substrate on both platforms, and its default rollback journal or write-ahead log gives the needed atomicity for a single-row insert. The commit itself costs a filesystem sync, which is a millisecond-scale operation on flash storage and worth every bit of it, because it is the difference between a lost job and a durable one.

The state machine on disk

A row moves through a small set of states, and every transition is itself a committed write. A workable set:

ENQUEUED -> RUNNING -> SUCCEEDED
                    -> FAILED (retryable, back to ENQUEUED with attempt + 1)
                    -> FAILED (terminal, no further attempts)
ENQUEUED -> CANCELLED

The transition into RUNNING is the one that needs care, because a process can die between the write and the completion. If the row is left in RUNNING after a crash, the next scheduler pass must decide what it means. Two policies are possible, and the choice is a trade rather than a correctness question.

Marking the row back to ENQUEUED on startup assumes the work did not complete. It is safe only if the operation is idempotent, because the work may in fact have completed before the crash and the row never got updated. Marking it SUCCEEDED assumes the work did complete, and silently loses work that did not.

The practical resolution is to make every operation idempotent and to assume non-completion. An upload keyed by a client-generated identifier is idempotent: a repeated request with the same key either replaces or is rejected, and the result is the same. A row insertion keyed by a primary key derived from the work identifier is idempotent for the same reason. Work that cannot be made idempotent needs a completion marker written by the receiver, which moves the problem to the far side of the network.

-- Lease a row for execution. The conditional update is the lock:
-- two schedulers racing here produce exactly one winner.
UPDATE jobs
   SET state = 'RUNNING',
       lease_until = strftime('%s','now') + 300,
       attempts = attempts + 1
 WHERE id = :id
   AND state = 'ENQUEUED'
   AND attempts < :max_attempts;

-- On startup, reclaim rows whose lease expired without a completion.
UPDATE jobs
   SET state = 'ENQUEUED'
 WHERE state = 'RUNNING'
   AND lease_until < strftime('%s','now');

The lease is not decoration. Without a lease, a row left in RUNNING by a killed process is indistinguishable from a row being executed right now, and startup recovery would either duplicate live work or skip dead work. The lease converts “is it running?” into a question the database can answer with a timestamp.

The enqueue side is where duplicates are born

A single logical action can produce more than one row. The user taps twice; the app is restored from a saved state and replays an intent; a retry loop enqueues again after a timeout that the server had in fact already satisfied. None of these are database bugs, and all of them produce duplicate work.

Two mechanisms contain the damage. The first is a uniqueness constraint on a key derived from the logical action — the target resource plus the operation plus a client-chosen nonce — so that a second insert fails rather than adding a second row. The second is a policy for what to do when a duplicate is detected: replace the existing row’s parameters, keep the existing row and discard the new one, or append. Keeping is the usual choice for work that is already running, because replacing parameters mid-execution produces a run that matches neither the old nor the new description.

Cancellation needs the same care. Deleting the row while a worker holds the lease does not stop the worker, which is already inside a network call with no mechanism to observe the deletion. The reliable form is a flag the worker checks at its own boundaries, plus a rule that a cancelled row’s result is discarded if it arrives. On the platform schedulers this appears as an explicit cancellation signal delivered to the running task, which the task must poll or await rather than assume.

Why the platform scheduler cannot be the only record

Both platforms offer a system-level deferred-work scheduler. Android’s WorkManager persists work in its own database and guarantees execution across reboots; iOS’s background task scheduler asks the system to launch the app later for a registered identifier. Using them removes the need to write a durable queue, but not the need to think about durability.

The reason is that the platform scheduler owns the row, not the semantics. If the app’s own state says the work is done and the platform scheduler still has a pending request, the request will run and must be a no-op. If the app’s state says the work is pending and the platform scheduler has dropped the request — because the app was force-stopped, because the identifier was never re-registered, because the request expired — nothing will run until the app notices and re-enqueues. On iOS the registered identifiers are not persisted by the system across launches in the way an app database is; they must be submitted again, and a launch that skips that step silently loses the schedule.

The practical pattern is two records with different owners: the app’s database holds the authoritative state of every logical unit of work, and the platform scheduler holds a hint that there may be something to do. Reconciliation runs on every launch and re-submits hints for rows that are not terminal. The hint is expendable; the row is not.

That reconciliation is also where constraints get applied, and constraint matching is the part of the scheduler that most often surprises: a job declared to require an unmetered network does not run on a metered one no matter how overdue it is, and a job declared to require the device to be idle may wait a long time on a device that is never idle. The constraint is a correctness statement about the work, so weakening it to make the job run sooner changes what the job means. See Timer Coalescing and the Deferrable Work Budget for how the system decides when an eligible job actually runs.

Migrating the record forward

A row written by version 3 of an app is read by version 4. The parameters it carries were meaningful under the old schema, and the operation it names may have been renamed, split, or removed. The scheduler therefore has a schema of its own to migrate, and the migration runs at a moment when the process may be killed mid-way.

Three rules keep this tractable. Migrations are numbered and applied in order inside a transaction, so a partially applied migration rolls back. Unknown job kinds encountered during a pass are dropped rather than retried, because a retry of something the current build cannot interpret will fail forever and hold a slot. Rows that have exceeded their attempt budget are moved to a terminal state rather than deleted, so that the failure is visible to whatever diagnostic surface the app has; the budget itself is arithmetic that the scheduler does not perform on the app’s behalf, as Retry Accounting When the Scheduler Chooses the Moment sets out.

The uncomfortable case is a job whose parameters are no longer valid — a file path that the new version stores elsewhere, an account that has been signed out. The row cannot be executed and cannot be silently dropped without losing a user-visible effect. Recording the failure and surfacing it is the only honest option; retrying it is a loop that never converges.

Boundary conditions

An empty queue is not the same as a completed queue. A launch that finds no rows cannot distinguish “everything finished” from “the database was never created” from “the database was deleted with the app’s data”. Anything that reports completion to a server must derive it from a record it wrote, not from the absence of pending rows.

Clock changes affect leases. A lease computed from wall-clock time is wrong after a manual time change or a timezone database update that moves the offset; a lease computed from a monotonic source does not survive a reboot. The workable compromise is to store both and treat the row as expired if either comparison says so, accepting an occasional premature reclaim in exchange for never leaving a row stuck.

Storage pressure is the last boundary. A device that is nearly full can fail the commit that was supposed to make the work durable, and the failure arrives as an exception from the database layer at the point where the app believed it was safe. Catching that exception and reporting success is worse than the exception, because it converts a detectable failure into a silent loss.