Event Sourcing and the Append-Only Log as System of Record

A conventional store keeps the current state of an entity and overwrites it. The history of how it reached that state is either discarded or reconstructed from an audit table that is written separately, often with different semantics, and frequently with gaps where a bulk update bypassed it. Event sourcing inverts the arrangement: the appended sequence of state-changing facts is the record, and the current state is a function computed from that sequence. There is nothing to overwrite, so there is no lost update to detect after the fact.

The change is not a storage format. It moves several things that are implicit in a mutable store — the atomicity of a state change, the meaning of a version, the definition of a snapshot, and the lifetime of a schema — into the open, where each has to be decided explicitly.

A write is an append guarded by a version

Saving an aggregate means appending the events that the command produced, conditioned on the aggregate still being at the version the command read. The condition is what replaces row locking or a compare-and-set on the state: if another writer appended in between, the expected version no longer matches and the append is rejected.

-- the version is part of the key, so two appends cannot claim the same one
CREATE TABLE events (
    stream_id   text        NOT NULL,
    version     integer     NOT NULL,
    type        text        NOT NULL,
    payload     jsonb       NOT NULL,
    recorded_at timestamptz NOT NULL DEFAULT now(),
    PRIMARY KEY (stream_id, version)
);

-- a stale expected version inserts nothing; the zero update count is the conflict
INSERT INTO events (stream_id, version, type, payload)
VALUES ('order-b41f', 7, 'OrderConfirmed', '{"totalCents":4200}')
ON CONFLICT (stream_id, version) DO NOTHING;

The version is an integer per stream, and the store enforces uniqueness on the pair of stream identifier and version. That constraint is the entire concurrency control mechanism: two appends cannot both claim version 7. A handler that loses the race has to re-read the stream, re-run the command against the new state, and retry — which means command handlers must be free of unrepeatable side effects. An email sent before the append is not undone by the retry.

This is a stronger guarantee than an update that overwrites the whole row, and a weaker one than a serializable transaction across several aggregates. Multi-aggregate commands need a process that records its own progress, not a larger transaction.

Replay, snapshots, and the cost of a long stream

Loading an aggregate means reading its events and folding them. A stream with ten events is cheap; a stream that has accumulated thousands makes every command pay for the full history, and the latency of a command grows with the age of the entity rather than with the work it performs.

A snapshot is the answer, and its definition must be precise: a snapshot is a cached fold of all events up to and including a recorded version, stored separately from the log and never authoritative. Loading reads the newest snapshot, then applies only the events after its version. If the snapshot is corrupt, or the fold function changed, discarding it must be safe and must produce identical state from the log. A snapshot that cannot be discarded has quietly become the record.

Snapshot frequency is a trade between write cost and load cost. Writing a snapshot every k events adds one write per k appends and caps the events replayed on load at k plus the snapshot’s own read. The right k depends on the fold’s cost per event rather than on the event count alone, and it often differs per aggregate type: an aggregate whose events replay a heavy calculation needs snapshots sooner than one whose events assign fields. Snapshotting every event is a mutable store with extra steps; snapshotting never is a log whose read cost grows without bound.

Two orderings, and they are not interchangeable

An event store provides per-stream order, which is a consequence of the version integers, and it usually provides a global order across streams, exposed as a monotonically increasing position assigned at append time. Projections consume the global order; aggregates load by stream.

The distinction matters because the global position says nothing about causality between aggregates. Two events at positions 100 and 101 may concern unrelated aggregates, and an event for aggregate A may depend on a decision that was made concurrently with an event for aggregate B. A projection that needs cross-aggregate consistency must be written to tolerate any interleaving the global order permits, which usually means deriving it from the events rather than assuming the append order reflects a business sequence.

Global positions are also the checkpoint for subscribers, and a subscription that resumes from a stored position can replay an event that was already applied. The projection’s apply step must be idempotent for that reason, the same requirement that CQRS read models impose.

Events are immutable, so their schemas are permanent

An event that has been written is data. Changing the meaning of a field retroactively changes what every past fold produces, and a stored aggregate’s state would silently differ depending on when it was loaded relative to the change. The constraints that follow are stricter than in a codebase where the schema changes with a migration.

Adding a field is safe when readers treat absence as a defined default. Removing a field is safe only after every reader no longer needs it, which for a log that may be replayed from the beginning is never. Renaming is a removal plus an addition. Changing a field’s type or unit is the dangerous case, because the old and new values are indistinguishable in the log; it requires a new event type or a new field, with a rule for which one wins when both are present.

Because stored events cannot be rewritten in place, evolution happens at read time. A loader can translate an older event shape into the current shape before folding, and that translation is code with the same test burden as a migration. The alternative, rewriting the log, is available only if the log is treated as derived data, which contradicts its role as the record.

Serialization ties into this. A format that omits absent fields and tolerates unknown ones, such as JSON with an explicit schema registry or a format with tagged fields, allows a reader to ignore what it does not understand. A format that requires the full shape rejects old events outright, and the log becomes unreadable after the first added field.

What the log cannot do by itself

Three practical constraints come with the record being append-only.

Deleting data is the first. A record that must be erased — a subject’s personal data, a payment detail that regulations forbid retaining — cannot be removed from an append-only log without rewriting it. The workable techniques are to store the sensitive value outside the event and keep only a reference in the log, deleting the referenced value, or to encrypt per subject and discard the key. Both leave the event structure intact and make the erased value unrecoverable, which satisfies the intent while preserving the log’s integrity. Neither is a deletion in the literal sense, and the distinction has to be documented for anyone auditing the store.

Bulk reconstruction is the second. A new projection reads the whole log, which costs time proportional to the total number of events rather than to the size of current state. A system that has been running for years cannot rebuild a projection from scratch in an interactive window; it needs a bounded replay, a snapshot of the log itself, or a migration path that starts the new projection from a point in time.

The dual-write problem is the third. Appending an event and publishing it to a broker are two operations, and a process that fails between them either loses the publication or duplicates it. Writing the publication as part of the append, or reading the log directly as the broker, removes the gap. A separate publish step with a retry has to accept that consumers may see an event twice and that a crash may delay one indefinitely.

Where the style is worth its cost

The style fits domains where the history is itself the business: ledgers, inventory movements, order fulfilment, anything where “what happened” must survive an audit and where the current state is a convenience. It fits poorly where the state is a document edited continuously — a wiki page, a profile — because the events become field-level diffs and the log is harder to read than the state it produces. In between are systems that use it for a subset of aggregates and a conventional store for the rest, with the log feeding the rest through the event-notification boundary.

The benefit that justifies the machinery is not audit alone. Every state the system has ever had is addressable: a fold to any version yields the aggregate as of that point, so temporal queries and retroactive correction become operations on the log rather than reconstructions from fragments. The cost is that every one of those states must remain derivable, which is what makes schema discipline the central engineering problem.