The Durable Outbox: Committing Locally Before the Network Agrees

The naive mobile write is a request sent from the UI, with the local view updated on success. It fails in a way that is easy to overlook during development on a reliable network and impossible to overlook in use: the request is sent, the process is killed before the response arrives, and the record of the intent is gone. The user saw a spinner, then nothing, and the server has no idea a write was attempted.

The durable outbox inverts the order. The intent is written to local storage first, in the same transaction that updates the local view, and only then is it transmitted. The local store is the source of truth for what the user has done; the network is a replication channel that may be slow, may fail, and may be absent. Everything else in offline behaviour follows from that inversion.

Writing the intent, not the request

The first design decision is what to store. Storing the serialised HTTP request is tempting because it can be replayed verbatim, and it is wrong for three reasons.

A serialised request embeds the URL, headers, and body as they were at the moment of enqueueing. If the server’s API changes, or if the token that authorised the request expires before the queue drains, the stored bytes are no longer valid and cannot be adjusted. A queue full of expired requests is a queue that fails on every drain attempt.

A serialised request also embeds the assumption that the endpoint will exist. An outbox entry that names an endpoint has coupled durable local state to a remote contract, and the coupling is invisible until the contract changes.

Storing the intent, meaning the operation and its arguments in the application’s own terms, keeps the translation to a request at drain time, when the current token, the current endpoint, and the current schema are all available. The entry says “set the quantity of item 4821 to 3”, and the drain code decides how to say that to the server today.

The cost of this choice is that the drain code must be able to interpret every operation type that might be sitting in the queue, including operations enqueued by a previous version of the application. An application that changes its operation vocabulary must migrate the queue, or keep interpreters for old operations, and neither is free. The alternative, storing bytes, merely moves the versioning problem into an opaque blob where it is harder to see.

Atomicity with the local view

An outbox that is written in a separate transaction from the local view creates a window where one exists and the other does not. A crash in that window leaves the user’s screen and the queue disagreeing, and no later code can tell which is correct.

The fix is to write both in one transaction, which requires them to live in the same store. A relational store makes this natural: the row representing the item and the row representing the queued operation are inserted in the same transaction, and the commit is the moment the user’s action becomes durable.

BEGIN IMMEDIATE;
UPDATE items SET quantity = :qty, dirty = 1 WHERE id = :id;
INSERT INTO outbox (op, entity, payload, created_at, seq)
VALUES ('set_quantity', :id, :qty, :now, :next_seq);
COMMIT;

BEGIN IMMEDIATE takes the write lock at the start rather than at the first write, which avoids a deferred transaction upgrading its lock later and failing with a busy error under concurrency. The dirty flag on the item is what lets the local view show unsent state without consulting the outbox for every render.

The same structure works with a key-value store that supports atomic batches, but only if the batch can span both the item and the queue entry. A store that offers per-key atomicity and nothing else forces the two writes apart, and the window returns.

The outbox should be an append-only log with a monotonically increasing sequence number, not a table that is updated in place. Sequence numbers give a total order, which is what the drain needs to preserve causality: an update to an item followed by a delete of the same item must be sent in that order, or the server will see a delete followed by an update to something that no longer exists.

Draining without duplicating

The drain is a single worker that reads the oldest entry, translates it to a request, sends it, and deletes the entry on success. The interesting part is the failure between sending and deleting.

If the process dies after the server processed the request but before the entry was deleted, the entry survives and is sent again. The outbox has therefore converted a possible loss into a possible duplicate, which is the correct direction, because a duplicate can be detected and a loss cannot be recovered. Detection requires the server to recognise the repeat, and the mechanism is the idempotency key described in Retry classification when the failure arrives as a timeout.

The key must be generated when the entry is created, stored in the entry, and sent unchanged on every attempt. Deriving it from the sequence number and a device identifier is convenient, because both are already in the entry and both survive process death. A random key generated at send time is useless, because a resend generates a different one.

The delete-after-success ordering has a second failure mode: the delete may succeed while the server’s response is lost. That is the same case as above, and the same key handles it. What must not happen is deleting before sending, which loses the operation entirely if the send fails, and that is why the entry is deleted only after a response the client can classify as success.

Ordering and head-of-line blocking

A strict FIFO drain is the simplest correct ordering, and it has a failure that users notice: one entry that cannot succeed blocks everything behind it.

An entry can be permanently unsendable: a reference to an item the server deleted, a value the server rejects, a permission the user has lost. With a strict queue, every later entry waits behind it forever, and the outbox grows while the application appears frozen in a partially-synced state.

Three resolutions exist, and they are not equivalent.

Skipping the failed entry preserves progress and breaks ordering. If the blocked entry was an update that a later entry depends on, skipping produces a server state that never existed locally, and the discrepancy may be unresolvable without a full reconciliation.

Quarantining the failed entry, by moving it to a separate table after a bounded number of attempts and continuing, has the same ordering break, but makes the break visible and auditable. The quarantined entries can be surfaced to the user as unsent changes, which is at least honest.

Refusing to proceed until the entry is resolved preserves correctness and requires a resolution path: the entry must be inspectable, correctable, and discardable by the user. This is the design that offline-capable note and mail applications use, and its cost is a UI that must expose the queue.

The choice depends on whether the operations commute. Two updates to different items commute and can be sent in any order. Two updates to the same item do not, and the queue must not reorder them. An outbox can encode this by grouping entries that touch the same entity into a chain that is drained in order, while allowing chains for different entities to proceed independently, which trades the simplicity of a single queue for the ability to make progress past one bad entity.

Compaction

A queue that grows with every keystroke is a queue that sends more than it needs to. If the user adjusts a quantity five times while offline, the server needs the final value, not the sequence.

Compaction collapses entries that are superseded. The rule must be defined per operation type, and it must be conservative: two set_quantity operations on the same item collapse to the last one, but a set_quantity followed by an add_item do not, and a set_quantity followed by an increment collapse only if the increment is applied to the value that the set produced. Getting this wrong silently drops a user’s action, which is worse than sending redundant requests.

The safe version of compaction operates only on entries that have not been attempted, only within one entity, and only for operations whose semantics are declared to be last-write-wins on the same field. Everything else stays. In practice this covers the high-frequency operations such as toggling a flag, adjusting a value, or moving an item, and leaves the low-frequency ones alone, which is where the queue’s volume comes from anyway.

The states a user can see

An outbox introduces a third state between “saved” and “failed”, and the interface has to represent it or the user will infer something wrong.

An entry that is queued and not yet sent is pending. An entry that has failed at least once and is awaiting a retry is retrying, and the difference from pending matters because it may persist for minutes. An entry that has exhausted its attempts or been quarantined is failed, and it needs a user-visible path to resolution.

The local view should render from the local store, with pending and retrying entries applied optimistically, so the user sees their action take effect immediately. The risk of optimism is that the server may reject the change, and the local view must then roll back, which requires the store to keep enough information to reconstruct the pre-change state, or to re-fetch it. Rolling back an optimistic change after the user has built further changes on top of it is the hardest case in offline-first design, and the reason some applications restrict optimistic writes to operations the server will not refuse.

None of these problems has a solution that is both simple and complete. The outbox’s value is that it makes them explicit and localised: durability, ordering, duplication, and compaction are all decisions about one table, visible in one place, rather than assumptions scattered across the call sites that send requests. That is what makes the offline behaviour of an application reviewable at all, and it is the foundation that Replaying a queue against state that has moved on builds on.