Why Exactly-Once Delivery Stops at the Workflow Boundary

A stream processor can offer exactly-once processing because it controls both ends of the pipe: it reads the source, it owns the state, and it writes the sink, and it can commit all three together. A workflow scheduler controls only the middle. It does not own the source, it does not own the sink, and it cannot commit anything on their behalf. That structural fact, not an implementation shortcoming, is why the guarantee has to be reconstructed from parts when the producer and consumer of the data are separate scheduled tasks.

Three participants, two commits

End-to-end exactly-once means each input record’s effect appears in the output exactly once, even when any participant crashes and restarts. Achieving it requires that three things happen atomically: the input is marked as consumed, the output is made visible, and the state that ties them together is advanced.

In a single process with a transactional source and a transactional sink, one commit can cover all three if the source and sink share a transaction coordinator, which is what makes the guarantee available in systems built as one runtime. Where the coordinator spans two systems, the commit becomes a two-phase protocol, and the protocol’s blocking window is the price. Where the systems do not share a coordinator at all, no atomic commit exists, and the guarantee must be assembled from idempotence plus at-least-once delivery.

The scheduler adds a fourth participant with no transaction at all. A task in a workflow is an external process that the scheduler starts; the scheduler records that it started, and later records what the process reported. Between those records the task’s effects on the source and sink are invisible to the scheduler, and the scheduler’s own state is not in the same transaction as either.

The ordering that a workflow imposes

A scheduled task typically reads from a source with an offset or a partition marker, writes to a sink, and reports success. Three orderings are available, and each has a distinct failure mode.

Write the sink, then commit the offset. A crash between the two leaves the output written and the offset unadvanced, so the retry reprocesses and writes again. The result is at-least-once, and the duplicate is in the output. This is the ordering most tasks implement, often without deciding to, because the sink write is the task’s own statement and the offset commit is a separate call at the end.

Commit the offset, then write the sink. A crash between the two leaves the offset advanced and the output missing, so the record is never processed. The result is at-most-once, and the loss is silent, because the offset says the work is done. This ordering is rare in deliberate designs and common in ones where the offset commit happens as a side effect of the read, which is the behaviour of a consumer that auto-commits on a timer.

Write the sink inside a transaction that also commits the offset, if the source and sink share a coordinator. This is the only ordering that is exactly-once without further assumptions, and it requires both ends to expose transactions that the task can join. Most batch sources, including a filesystem, a table partition, and an object store prefix, expose no such transaction, so the option is unavailable for the common case of a task that reads a partition and writes a table.

Idempotence as the substitute for a transaction

Where the transaction is unavailable, the achievable guarantee is at-least-once delivery combined with a sink that absorbs duplicates. That combination is often described as effectively-once, and the qualifier is doing real work: the guarantee holds only for the effects the sink can deduplicate, and only while the deduplication key is correct.

The key has to be derivable from the input, not from the attempt. A record identifier from the source, a composite of business columns, or a hash of the record’s content all qualify. The attempt identifier does not: it changes on every retry, so deduplicating on it deduplicates nothing. This is the distinction between a ledger of applied attempts, which prevents a second application of the same attempt but not of the same record by a different attempt, and a ledger of applied records.

The sink must apply the key in the same atomic operation as the write. An upsert on a primary key does this; a delete-then-insert inside a transaction does this; a check followed by an append does not, because the check and the append can be separated by a crash. For append-only sinks with no key, the ledger of applied record identifiers is the only route, and it grows with the data rather than with the runs.

The limits of the substitute are worth stating precisely. It fails when the input is not uniquely keyed: a source that emits the same logical event twice with different identifiers, or events with no identifier at all. It fails when the transformation is non-deterministic, because two attempts on the same record produce different outputs and the sink’s deduplication keeps whichever arrived first. And it fails when the effect escapes the sink: a task that both writes a table and sends a message has one effect the sink can deduplicate and one it cannot.

Where the scheduler’s retry lands

The scheduler retries a task, and the retry is a new process with no memory of the previous one. Whatever the previous attempt did is not visible to it except through the state it left in the source and sink, and that state is precisely what the retry is supposed to handle.

This is why the scheduler’s retry count and the task’s internal retry logic must not be layered without thought. A task that internally retries its sink write three times, wrapped in a scheduler retry of three attempts, produces up to nine attempts at the write, and the amplification applies to every idempotence mechanism the sink has. A sink that deduplicates correctly absorbs them; a sink that does not receives nine copies and reports one success.

The scheduler also cannot tell the difference between a task that failed before its write and one that failed after it. Both look like a non-zero exit. If the task’s exit code distinguished them, with a specific code for “failed, nothing committed” versus “failed, possibly committed”, the scheduler could retry the first freely and apply a stricter policy to the second. Almost no scheduler models this, so the task’s exit status collapses the two, and every retry must be treated as the dangerous case. That collapse is the practical reason idempotence is a requirement rather than an optimisation: it is the only property that makes the collapse safe.

What a transactional sink changes

A sink with a transactional log changes the calculus, because the commit becomes a single metadata operation with a known version, and the task can be written to commit conditionally on that version. The task then has a precise notion of “my output is visible” when the commit succeeded, and “my output is not visible” when the commit failed. The gap between them shrinks to the commit operation itself, and the commit is the only window in which an ambiguous outcome is possible.

Two attempts committing the same logical interval now conflict, and the log rejects one of them. Which one is rejected depends on the condition: a commit asserting the prior version succeeds for the first and fails for the second, so the loser knows it lost. A commit that overwrites unconditionally succeeds for both and leaves the second’s content, which is last-writer-wins rather than exactly-once.

The version assertion is the same mechanism as the fencing token in worker leases and zombie tasks, applied at the commit rather than at the lease. The two are complementary: the lease bounds how long the scheduler waits, the version assertion bounds which attempt may commit. Neither alone gives exactly-once, and together they give exactly-once for the sink’s effects, which is the part of the guarantee that a workflow can actually own.

What remains outside is the source. A task reading a partition cannot mark that partition as consumed in the same transaction as its sink commit unless the source supports it, and most batch sources do not. The result is that a retry re-reads the same input, and the sink’s deduplication must handle the re-read. For a partition that is immutable once written, this is harmless: the re-read produces the same records, and the same keys are deduplicated. For a partition that can change between attempts, the retry may read a different set of records, and the output becomes a function of when the attempt ran rather than of the interval it claims to cover. Pinning the input, by reading a fixed version, a fixed file listing, or a snapshot identifier, restores determinism at the cost of requiring the source to offer versions.

The guarantee that is actually available

For a scheduled workflow, the honest statement is: at-least-once execution with effectively-once effects at each sink that can deduplicate on a stable key, provided the task’s transformation is deterministic and its input is pinned. Every clause is load-bearing, and each names a system the scheduler does not control.

The alternative, a workflow engine that offers exactly-once directly, exists only when the engine also owns the data path, which is the case for a stream processor and not for a scheduler that dispatches external processes. Recognising that boundary is more useful than trying to close it, because it tells where the effort belongs: on the sink’s commit protocol, on the task’s key derivation, and on pinning the input, all of which are properties of the data systems rather than of the graph. The graph’s contribution is narrower and still essential: it ensures a task runs when its inputs are ready and records what happened when it did, which is the substrate every one of those mechanisms rests on.