Compensating Transactions in Multi-Service Workflows

A business operation that spans three services and takes minutes cannot be a transaction. It can be a sequence of local transactions, each durable on its own, with an explicit plan for what to do when a later step fails and the earlier ones must be undone. That plan is a saga, and the undo actions are compensations.

The word “undo” is misleading in a way that causes real design errors. A compensation is not a rollback of a database write. It is a new business action with its own side effects, its own possibility of failure, and its own audit trail. A refund is not the absence of a charge.

A transaction that lasts longer than a request

The classic formulation comes from Garcia-Molina and Salem’s 1987 paper, which defined a saga as a sequence of transactions T₁ through Tₙ with compensating transactions C₁ through Cₙ₋₁. If every step completes, the saga succeeds. If step k fails, the compensations for steps k−1 down to 1 run in reverse order, and the saga is either fully applied or fully compensated.

The long duration is what forces the structure. Holding a database transaction open across a human approval, a payment authorisation, or a supplier’s confirmation would hold locks for minutes, and no database is designed to hold row locks across an unbounded wait. Even where a distributed transaction protocol is available, its blocking behaviour makes it unusable for this shape of work.

The steps are local transactions in the participating services. Each one commits, becomes visible, and stays committed unless a compensation reverses it. That is a weaker guarantee than atomicity, and the weakness is not in the failure handling but in the visibility of intermediate states.

Compensations are new actions

A compensation has to be defined for each step that can be undone, and the definition needs the same care as the forward action.

A reservation is compensated by releasing it, which is idempotent and cheap. A charge is compensated by a refund, which is a second movement of money, visible on a statement, subject to its own settlement delay, and possibly refused if the original transaction has already been disputed. An inventory decrement is compensated by an increment, which may fail if the item was sold to someone else in the interval, leaving the count inconsistent in a way the compensation cannot repair.

Some steps cannot be compensated at all. An email that has been delivered, a physical parcel that has been handed to a carrier, a notification sent to a partner’s system, a record written to an append-only audit log. For these, the only options are to place them after the point of no return, or to accept that the workflow has a semantic failure mode the software cannot fix.

This leads to a classification that drives the ordering of steps:

Step class	Property	Placement
Compensatable	Can be undone by a defined action	Before the pivot
Pivot	The point of no return	Central, exactly one
Retryable	Guaranteed to succeed once reached	After the pivot

The pivot step is the one where the workflow becomes committed in the business sense. Everything before it can be undone, everything after it must eventually succeed, so retryable steps are retried until they do and never compensated. Putting a non-compensatable step before the pivot produces a workflow that can fail with no recovery path, which is the commonest structural mistake in saga design.

No isolation and what it exposes

Sagas sacrifice atomicity and isolation. Atomicity is replaced by compensation; isolation is simply gone.

Intermediate states are visible to every other reader. An order exists with status “awaiting payment” while the payment is in flight, and a report that counts orders by creation time will include it. A read model built from events will show it. Another request that reads the order and decides based on its status must know that the status is provisional, which is one of the freshness guarantees a projection has to publish, as set out in Read Models Across Service Boundaries.

The subtler failure is a write skew between two sagas. Two workflows each read a shared value, each decide their step is valid, and each write, and the combination violates an invariant that neither individual step broke. A discount campaign that checks a budget and then spends it has exactly this shape when two campaigns run concurrently.

Four countermeasures are in common use, none complete:

A semantic lock marks records as being modified by a saga, so other operations either wait or refuse. The order carries a status that makes it ineligible for the operations that would conflict. This works when every writer respects the status, which requires the status to be part of the invariant rather than a display field.

Commutative updates make concurrent steps safe by construction. Incrementing a counter and decrementing it are commutative, so the order of the two does not matter. This works only for operations that are genuinely commutative, which excludes most decisions based on a threshold.

Re-reading before writing narrows the window without closing it. A conditional update that checks a version, or a database constraint that rejects the invalid combination, converts the race into a failed step that the saga can retry or compensate. This is the practical option for most cases, and it moves the enforcement into the same place as the local transaction.

Serialising the conflicting steps through a single owner removes the concurrency entirely, at the cost of throughput and of a participant that all sagas must route through.

Orchestration and choreography

The saga logic has to live somewhere, and there are two arrangements.

An orchestrator is a service or a component that holds the workflow definition, sends commands to participants, waits for replies, applies timeouts, and decides when to compensate. The state machine is explicit and inspectable, timeouts are enforced in one place, and a stuck workflow can be diagnosed by reading one row. The cost is an extra service in the path, a place where the workflow’s logic accumulates, and a component that must be highly available because it holds the state that makes recovery possible.

Choreography has no coordinator. Each participant listens for events and decides its own next action, and the workflow exists only as the aggregate of those decisions. There is no central bottleneck and no orchestrator to fail, but the workflow is emergent: no single file describes it, adding a step requires changing several services, and cyclic dependencies between events are easy to create and hard to detect. Timeouts have to be owned by whoever is waiting, and there is no natural place to record that a saga is stuck.

Most systems with more than a handful of steps converge on orchestration for the complex flows and choreography for the simple ones. The split is fine as long as the simple flows stay simple.

type Step struct {
    Name       string
    Compensate func(ctx context.Context) error // nil for retryable steps
}

// Compensate runs in reverse order and stops at the first error that
// cannot be retried; the remaining steps stay recorded for an operator.
func Unwind(ctx context.Context, done []Step) error {
    for i := len(done) - 1; i >= 0; i-- {
        if done[i].Compensate == nil {
            continue
        }
        if err := done[i].Compensate(ctx); err != nil {
            return fmt.Errorf("compensating %s: %w", done[i].Name, err)
        }
    }
    return nil
}

Durable state and the stuck workflow

An orchestrator that keeps saga state in memory loses every in-flight workflow when it restarts. The state therefore lives in a database: the current step, the inputs collected so far, the deadline, and the compensation plan. Recovery reads the records and resumes from the last committed step, which requires every step and every compensation to be idempotent, because a crash between the external action and the state update produces a replay.

The state record should outlive the workflow. Retaining completed and compensated sagas for a bounded period, often thirty days for financial flows, answers the question that arrives later: why was this order cancelled, and which step failed. Without the record, the answer exists only in logs that have already rotated.

Compensation failure is the case that separates a design from a demonstration. A refund that the payment provider rejects, a reservation release that times out, an inventory increment that conflicts with a concurrent sale. Each leaves the saga in a state that is neither complete nor compensated. The system needs a durable record of that state, an alert that names the specific saga, and a defined manual action, because automatic retry against a permanent error will never succeed and will hide the problem in a retry log.

The approach that avoids most of these cases is to make the compensated steps as small and as local as possible, and to place anything irreversible at the end of the sequence, after the last point at which the workflow can decide to abort. The Dual-Write Problem and the Transactional Outbox supplies the reliable messaging that lets each step and its notification commit together; the saga sits above that layer and cannot repair a message that was never sent.