Window Assignment and Trigger Semantics

A windowing operator answers two questions that are easy to confuse. Which windows does a given event belong to, and when does a window’s result get emitted. The first is a pure function of the event’s timestamp and the window definition; the second is a policy over watermarks, counters, and timers. Systems that fuse the two into one concept make the common case work and the unusual case impossible, because the emission policy that suits a live dashboard is wrong for a settlement job over the same data.

Assignment is a function of the timestamp

An assigner maps an event to one or more window identifiers derived from its event timestamp. The mapping ignores arrival order, which is what makes the operator’s output a function of the input’s content.

A tumbling window of size s assigns an event at t to the single window with index floor(t / s). Adjacent windows share no events, so the total state is one partial aggregate per window. A hopping window of size s and hop h with h < s assigns an event to every window whose start lies in (t - s, t], which is ceil(s / h) windows. A sliding window is the same construction with the hop expressed as a separate slide parameter rather than tied to the size.

The multiplication is the cost that surprises people. A one-hour window evaluated every minute assigns each event to sixty windows, so the operator keeps sixty partial aggregates per key instead of one, and each event causes sixty updates rather than one. Both the state size and the per-event work scale with s / h. For a key with a high event rate, that ratio is the difference between a job that fits in memory and one that does not, and the fix is usually to aggregate at a coarser hop and interpolate downstream rather than to run the fine hop over the raw stream.

Session windows break the pattern because their boundaries depend on the data. A session is a run of events separated by gaps shorter than a timeout; each event initially opens a window of its own, and two windows merge when an event or a gap closure proves they belong together. Merging is the operation that distinguishes session windows from the fixed-interval family: partial aggregates for overlapping windows must be combined, and the combined window’s identity is the earliest start of the merged set.

def assign(t: int, size: int, hop: int):
    # Windows whose start s satisfies  t - size < s <= t
    first = ((t - size) // hop + 1) * hop
    return range(first, t + 1, hop)

With size == hop this yields exactly one window; the formula degenerates correctly rather than needing a special case, which is a small argument for expressing tumbling windows as a hop of the same length.

Merging is where session windows cost

Session merging has a specific failure mode: an event that bridges two long sessions forces the operator to combine two large partial aggregates. If the aggregate is a sum, combining is one addition. If it is a distinct-count over a set of identifiers, combining requires a union of the two sets, and the set can be as large as the number of distinct values in the session. A pipeline that works on small sessions can fall over on one session with millions of distinct keys, and the state that must be held for the merge is proportional to the session length rather than to the event rate.

The mitigation is to choose aggregates that merge cheaply. Sums, counts, minima, and maxima are constant-size and merge in constant time. Approximate distinct counts over a fixed-size sketch merge by combining registers, which is the reason sketches are popular in streaming aggregations at all. Exact distinct counts are the case that cannot be made cheap, and a pipeline that needs them over long sessions is really asking for a batch job.

Triggers decide emission, not content

A trigger is a condition evaluated as events arrive and as the watermark moves, which decides that the current contents of a window should be emitted. It does not change what the window contains; it only decides when a snapshot of that content leaves the operator.

The watermark trigger fires when the watermark passes the window end. It is the default for event-time windows, and it produces exactly one emission per window when no record is late. A processing-time trigger fires on a periodic timer regardless of the data, which gives regular output at the cost of emitting partial results: a window that fires every ten seconds emits a sequence of growing totals.

An early trigger with a watermark trigger behind it gives the low-latency pattern: emit speculative results while the window is open, then emit the final result when the watermark passes. The speculative emissions are not wrong, they are incomplete, and a downstream consumer that sums them gets a number far too large. The distinction has to be carried in the record, either as an explicit “final” flag or by making each emission a replacement for the previous one.

A late trigger fires when a record arrives for a window whose watermark has already passed. It only has any effect if the window’s state is still retained, which is the allowed-lateness setting from Watermarks as Completeness Signals. Late firing is what converts a system that silently loses records into one that revises its answers, and the revision mechanism is Retractions and Updating Results.

Triggers compose. The standard combination is: early periodic emissions, a final emission on the watermark, and a late emission for anything that arrives afterward. The composition is what makes a single window produce a stream of records with the same key and window but different values, which forces every downstream consumer to have an opinion about revisions.

Accumulating versus discarding panes

When a window fires more than once, the operator must decide what the next emission contains. Accumulating mode keeps the partial aggregate and emits the running total, so successive emissions increase monotonically and the final one is the complete answer. Discarding mode clears the aggregate after each firing, so each emission covers only the events since the previous firing, and the complete answer is the sum of all emissions.

The choice determines what a consumer must do. Accumulating output is a sequence of replacements for the same window; discarding output is a sequence of disjoint additions. Mixing the two conventions in one pipeline produces double counting that is invisible in any single record. Accumulating mode is the better default because it makes each emission individually meaningful and makes the final emission the answer, and discarding mode exists for the case where the downstream is a sum and the operator wants to avoid retaining the whole aggregate.

Boundaries and the alignment question

Window boundaries are computed in event time, so a window of one hour starting at the epoch boundary covers 00:00 to 01:00 in whatever timeline the timestamps use. If the timestamps are UTC, the windows align with UTC hours; if they carry local offsets, an event at 23:30 with a +02:00 offset falls into a different absolute window than one at 23:30 UTC. The extractor’s job is to normalize to one timeline before assignment, and a pipeline that skips that step gets windows that drift with the producer’s locale.

The other boundary issue is the end of a window. A window is half-open, covering [start, start + size), so an event exactly at the boundary belongs to the later window. An implementation that closes intervals instead double-counts boundary events, which for a stream with timestamps at whole seconds is a systematic error rather than a rare one. The half-open convention is the one that makes adjacent tumbling windows partition the timeline exactly.

A final limitation is that windows over an unbounded stream never finish globally. There is always a last window open at the watermark, and its result is available only after the watermark passes it, which on an idle stream may never happen. A pipeline that must produce a result per window on a quiet source needs a trigger that does not depend on the watermark, and must accept that the result may later be revised.