Cron Triggers, Misfire Policy, and the Runs That Never Happen

A cron expression does not describe work; it describes instants. The work still has to be enqueued, claimed, retried, and completed exactly as any other job, and the scheduler’s whole contribution is deciding which instants produce a record. That decision is where the difficulty lives, because a scheduler is a process that can be down, can run in several replicas at once, and can be asked to fire a schedule whose previous run has not finished.

The five-field cron form inherited from crontab(5) expresses minute, hour, day of month, month, and day of week, with each field a set or a range or a step. The granularity is one minute, so a schedule has at most 1,440 firings per day, and the next firing after a given instant is computable by field comparison without iterating over every minute.

Recurrence as a stored next-fire time

The simplest scheduler is a loop: for each schedule, hold next_fire_at, and when the clock passes it, insert a job and compute the next value.

def due_schedules(now):
    # fetch schedules whose next_fire_at has passed
    rows = db.query(
        "SELECT id, expr, next_fire_at FROM schedules "
        "WHERE enabled AND next_fire_at <= %s ORDER BY next_fire_at LIMIT 100",
        (now,),
    )
    for row in rows:
        yield row


def advance(expr, scheduled_for):
    # strictly after the scheduled instant, so a retried advance is stable
    return next_firing(expr, scheduled_for + timedelta(minutes=1))

Two properties of that snippet matter more than the parsing. The advance is computed from scheduled_for rather than from “now”, so a scheduler that wakes late still produces the firing for the slot it missed and the next value is a function of the slot rather than of the wake-up time. And the scan is bounded, so a long outage that leaves many schedules overdue does not turn into one unbounded query.

The alternative is to compute firings from the clock at claim time and never store a next-fire value. That works for schedules with no history requirement but cannot answer whether a given slot already fired, which is the question catch-up and duplicate suppression both depend on.

Catch-up after downtime

A scheduler that was down from 01:00 to 05:00 has four missing hourly firings. Three policies are defensible, and they are not interchangeable.

Fire every missed slot preserves the one-job-per-period invariant. It is correct for a schedule whose jobs are independent and whose periods are semantically meaningful — an hourly aggregate that writes a row per hour, a billing tick that accrues per period. Firing four jobs at 05:01 produces four rows with their original timestamps, which is right, and four simultaneous executions, which may not be.

Fire once, for the most recent missed slot is correct when each run is a full recomputation rather than an increment: a nightly reconciliation that reads the whole table. Running it four times would do the same work four times, and only the last result matters.

Fire nothing is correct when the missed window is itself invalid — a trading-hours report generated at 09:00 is meaningless if generated at 15:00 — and the schedule should simply resume at the next valid slot.

The policy has to be per schedule, because the same system can contain all three cases. The mechanism is the same in each: the number of slots to emit is computed from the stored next_fire_at and the current time, bounded by a catch-up window. Beyond that window, the schedule fast-forwards to the next future slot and records that it skipped the rest. Without the bound, a schedule that was disabled for a month produces a month of jobs on the first tick, and those jobs compete with the current period’s work.

Emitting a burst of catch-up jobs at once is a load event. Their availability times should be spread over the following seconds or minutes rather than all equal to the tick time, or the scheduler has effectively converted a four-hour outage into a four-hour backlog delivered in one second. The promoted rows then enter the runnable set exactly as the future-dated jobs in Holding Delayed Jobs do, and the same claim contention applies to them.

One schedule, many replicas

Running the scheduler in more than one replica is the obvious way to avoid a single point of failure, and it immediately produces duplicate firings: every replica evaluates the same expression and inserts the same job.

Two mechanisms prevent it, with different failure characteristics. The first is leader election: a single replica holds a lock with a lease and is the only one that emits. The lock is a lease in the sense of Visibility Timeouts and Lease Heartbeats, and the failure mode is the same — a leader that loses the lock but has not noticed may emit for one more tick. The second is a unique constraint on the identity of the firing, (schedule_id, scheduled_for), with the insert using ON CONFLICT DO NOTHING. Every replica may evaluate, and only one insert succeeds.

The constraint approach is preferable when it is available, because it makes duplicates structurally impossible rather than unlikely and removes the election protocol entirely. It also makes the catch-up logic safe: two replicas computing the same missed slots cannot produce two jobs for a slot. Its cost is that every replica does the evaluation work, which is negligible, and that the unique key must exist — a schedule that fires twice per minute cannot use a minute-resolution key.

The reaper problem remains either way. A schedule’s next_fire_at must advance exactly once per emitted slot; if the advance is a separate statement from the insert, a crash between them either skips a slot or repeats it. Putting both in one transaction, or deriving next_fire_at from the maximum emitted slot rather than storing it, removes the window.

Calendar time versus elapsed time

A schedule like “every 24 hours” and a schedule like “daily at 02:00” look equivalent and are not. The first is elapsed time and drifts across every clock adjustment; the second is calendar time and moves with the local zone.

Cron expressions are calendar-time constructs, and the local zone is part of them. In a zone observing daylight saving, the hour that repeats at the autumn transition means a 01:30 schedule has two matching instants, and the hour skipped at the spring transition means a 02:30 schedule has none. Implementations differ in what they do with those: some fire once, some fire twice, some skip. RFC 5545, which defines recurrence for calendaring, handles the repeated hour by making the recurrence set depend on the local time interpretation, and skipping non-existent local times — but cron implementations are not required to follow it. The behaviour should be read from the documentation of the specific scheduler rather than assumed.

The consequence for job design is that a job must not depend on “once per calendar day” being exactly 24 hours apart. A daily aggregate that computes the difference between two timestamps and labels it a day will be wrong twice a year in a DST zone, and correct everywhere if the aggregation window is computed from local midnight boundaries rather than from a fixed interval.

Interval schedules have the opposite property: “every 15 minutes” is immune to DST and drifts against wall-clock alignment unless the next fire is anchored to a fixed origin. Fire times computed as “previous fire plus interval” accumulate error from every late wake-up; fire times computed as “next multiple of the interval after now” do not.

Schedules that must align to the wall clock — an hourly batch that downstream consumers expect at the top of the hour — should be staggered with a per-schedule offset. A hundred schedules all written as 0 * * * * produce a hundred jobs in the same second, every hour, which is a load spike with no functional purpose.

A run that overruns its period

A job taking longer than the interval between its firings produces overlapping executions. For an idempotent recomputation that is wasteful; for an append of a summary row it produces duplicates; for a job holding a resource — a table lock, a remote API quota — it can produce a deadlock or a rate-limit cascade.

Three policies cover it, and Kubernetes CronJob names them explicitly: allowing concurrency, forbidding it, and replacing the running job with the new one. Forbidding is the conservative default because it preserves the “at most one running” invariant at the cost of dropping periods; allowing preserves every period at the cost of unbounded pile-up; replacing loses the in-flight run’s partial work.

The choice interacts with the catch-up policy. With catch-up on and concurrency forbidden, a slow run causes the queued slots to be skipped rather than run once it finishes, so the schedule silently loses periods. With catch-up off, the missed periods are simply not represented. Either way, a run that regularly exceeds its period is a signal that the schedule is wrong — the job should be split, or its period extended — rather than a condition to be managed by dropping work.

Kubernetes also exposes a starting deadline: a run that has not started within that many seconds after its scheduled time is abandoned rather than started late. This is the direct expression of staleness: a job whose result depends on being fresh should refuse to run when it is old.

Detecting the run that never happened

A job that fails loudly is easy to notice. A schedule that stopped firing produces no errors, no failed jobs, and no traffic — it looks exactly like a system with nothing to do.

The scheduler cannot report its own silence, so the expectation has to live outside it. The common mechanism is a completion timestamp per schedule, written by the job itself rather than by the scheduler, and checked by a monitor that fires an alert when the timestamp is older than the schedule’s period plus a tolerance. Checking the presence of the output rather than the intent of the schedule also catches the case where the scheduler fired and the job silently did nothing.

Where the scheduler stores a heartbeat of its own — a row updated every tick — a monitor on that value catches a dead scheduler, but not a healthy scheduler with a broken schedule definition. Both checks are cheap, and they cover different failures.

The last edge is a schedule whose expression becomes invalid after an edit, matching no instant at all. A five-field expression with day-of-month and day-of-week both restricted matches the union of the two in the traditional cron semantics, which is rarely what the author intended, and an expression like the 31st of February matches nothing. A schedule that has a next-fire time of null should be rejected at write time, not discovered by the absence of its output months later.