Logical Dates, Interval Boundaries, and Catchup Semantics

A daily batch job is not really about the day it runs on. It is about the day whose data it processes, and those are different quantities that drift apart the moment anything goes wrong. Separating them, and deciding how the interval a run is responsible for is defined, removes a whole class of bugs that otherwise reappear every time the schedule is late, retried, or backfilled.

Two clocks

The wall clock is what the machine reads. The logical date is a label attached to a run by the scheduler, identifying which period the run is responsible for. For a job scheduled daily at 02:00, the run that executes at 02:00 on the 25th of March usually carries a logical date of the 24th, because it is processing the 24th’s data, the day that has just finished. That offset is the source of most of the confusion around these systems, and the confusion is worth resolving explicitly rather than leaving to convention.

The convention is a consequence of interval semantics. A daily schedule partitions the timeline into intervals: [24th 00:00, 25th 00:00), [25th 00:00, 26th 00:00), and so on. A run is responsible for exactly one interval. It cannot start until the interval has closed, because data belonging to the interval may still arrive before it closes. So the run for [24th, 25th) starts after the 25th begins, which means its start time is after its logical date. The logical date labels the start of the interval, and the interval’s end is the start of the next one.

Two properties follow. The first is that the label is stable: a run for the 24th carries that label whether it executes on the 25th, on the 30th during a backfill, or twice because the first attempt was wrong. Anything derived from the logical date, whether the partition it reads, the partition it writes, or the filename it produces, is therefore reproducible, which is the foundation of idempotent task boundaries. The second is that the label is not the run’s execution time, so any task that uses now() to decide what to process will disagree with the schedule as soon as the run is late.

Half-open intervals and the off-by-one

The interval [start, end) includes its start and excludes its end, which is the property that makes the cover disjoint. The alternative, [start, end], includes both endpoints, and then every instant at a boundary belongs to two intervals. A record timestamped exactly at midnight would be counted in both days’ runs, and the totals would exceed the true total by exactly the number of boundary records.

Half-open intervals make the partition a total, disjoint cover of the timeline: every instant belongs to exactly one interval, and the sum of the intervals’ outputs equals the output of the whole range. That property is what makes a backfill over a month equal to thirty daily runs, and it is what makes it safe to re-run a single day without re-running its neighbours.

from datetime import timedelta

def interval_for(logical_date, period=timedelta(days=1)):
    """Half-open [start, end): the run owns start, not end."""
    start = logical_date
    return start, start + period

def belongs(ts, start, end):
    return start <= ts < end      # 'end' excluded, so neighbours never overlap

The cost of the convention is that a source which reports by calendar day in an inclusive form must be converted. A source that provides “the 24th’s data” as everything with a date column equal to the 24th maps cleanly onto [24th, 25th). A source that provides a file covering 00:00:00 through 23:59:59 maps onto the same interval only if the last second’s records are actually the whole second; a source with sub-second timestamps that truncates at 23:59:59 silently drops the final second of each day. The truncation is in the source, but the scheduler’s interval definition is what makes it invisible.

Schedule time versus data time

The logical date is the schedule’s claim about what period a run covers. It is not a claim about what the data contains. A run for the 24th may read a source table whose latest partition is the 23rd because the upstream load has not finished, and it then produces an output labelled the 24th that contains the 23rd’s data. Nothing in the scheduler detects this, because the scheduler only knows that its own dependency, the upstream task, reported success.

Closing that gap requires a check on the data’s own time. Two forms exist. The first is a sensor or check task that asserts the expected partition exists before the run proceeds, which converts a missing input into a blocked run rather than a wrong output. The second is a watermark carried in the data itself, the maximum timestamp present in the source, compared against the interval’s end so that a run can refuse to proceed until the source has advanced past it. The second form is more precise and requires the source to expose such a value, which many sources do not.

The distinction matters most for late-arriving data. If records for the 24th arrive on the 27th, the run for the 24th has already completed and its output is missing them. There is no scheduling mechanism that fixes this after the fact; the run must be repeated, and the repeat must be safe, and whatever consumes the output must tolerate the output changing after it was first published. This is the reason the logical date has to be a first-class parameter rather than something inferred: it is the only handle by which a later correction can address the right interval.

Catchup as a bounded loop

When a schedule is paused, or the scheduler is down, or a new pipeline is deployed with a start date in the past, the intervals between the start date and now have no runs. Catchup is the policy that decides whether to create them.

With catchup enabled, the scheduler enumerates the intervals from the configured start date to the current time and creates a run for each. With it disabled, the scheduler creates a run only for the most recent closed interval and ignores the rest, so the pipeline begins operating from deployment time and never processes the history.

The enumerated form has a cost that grows with the size of the gap. A pipeline deployed with a start date two years in the past and a daily schedule creates seven hundred and thirty runs, each with its own tasks, and if the scheduler’s concurrency limits allow only a few at a time, the backlog takes as long as it takes. The runs are also competing with the live schedule for the same workers and the same warehouse capacity, which is the interaction described in backfill concurrency.

The disabled form has a different cost: the gap is permanent. Whatever the pipeline was supposed to produce for the missed intervals does not exist, and no later run will produce it, because each run only covers its own interval. The usual compromise is catchup disabled by default with explicit backfill for chosen ranges, so that the expensive operation is a deliberate act with a known scope rather than a side effect of deploying.

A third policy exists in some systems: catchup limited to the most recent n intervals, so that a short outage is repaired automatically and a long one is not. It bounds the work at the cost of an arbitrary boundary: a gap of n intervals is repaired and a gap of n+1 silently is not, and the difference between the two is invisible in the run history.

The end of the interval and the start of the run

The offset between logical date and execution time is also the offset between the interval’s close and the run’s start. A daily job at 02:00 has two hours of slack between the interval ending at midnight and the run beginning. That slack exists to let the source’s own load complete, and it is a data dependency expressed as a delay rather than as a dependency.

Expressing it as a delay is fragile: if the source load takes longer than two hours, the run starts against incomplete data, and the schedule has no way to know. Expressing it as a dependency, with the run waiting for the source load’s completion signal, is precise but couples the two schedules, and a failure in the source load now blocks the entire downstream chain rather than merely delaying it. Most systems end up with both: a delay as the default timing, and a check at the start of the run as the correctness backstop. The delay handles the normal case cheaply; the check handles the abnormal case correctly, and its failure is an ordinary task failure that the retry policy can treat like any other.

What no policy can do is make a schedule’s intervals align with a source that does not report on interval boundaries. A source that publishes whenever it finishes, at an unpredictable hour, cannot be partitioned by a daily schedule without either waiting for a signal or accepting a lag. The logical date is the scheduler’s model of time; when the data’s time does not fit that model, the model has to be relaxed rather than the data forced into it.