Data Engineering · Cluster
Stream Processing
Event time, watermarks, and state.
- 01 Event Time and Processing Time in a Stream Operator Why the clock an operator reads decides whether a replay reproduces its output, and how the two clocks diverge when a source falls behind.
- 02 Watermarks as Completeness Signals How a watermark asserts that no earlier event will arrive, how it is propagated through operators, and what happens when the assertion is wrong.
- 03 Window Assignment and Trigger Semantics How an event is mapped to windows, why sliding windows multiply state, and how triggers separate when a result is emitted from what it contains.
- 04 Measuring Disorder to Size a Lateness Budget How to derive the watermark delay and allowed lateness from the arrival-delay distribution, and why a percentile is the only honest input to that choice.
- 05 Keyed State Layout and the Cost of Keeping It What a keyed operator stores per key, how state backends trade heap for disk, and why a growing key space is the failure that has no trigger.
- 06 Interval Joins Over Event Time Why a stream-stream join must buffer both sides, how the watermark of the slower input gates the match, and where the retained state actually goes.
- 07 Two-Phase Commit and Exactly-Once Sinks Why a stream job cannot deliver exactly once without the sink participating, and how a prepare-then-commit protocol ties checkpoint state to external writes.
- 08 Checkpoint Barriers and Aligned Snapshots How a barrier injected at the source cuts a consistent snapshot across a distributed operator graph, and what alignment costs when one channel stalls.
- 09 Rescaling Keyed State Without Replaying History Why changing an operator's parallelism redistributes state, how the key-group indirection makes the move a file copy, and what forces a full replay instead.
- 10 Retractions and Updating Results How a stream of window results becomes a changelog, what a retraction record must contain, and why consumers must know the difference between a delta and a replacement.