Deadline Misses: Detection, Recovery, and Watchdog Design

A schedule that meets its deadlines in the nominal case will meet them until it does not, and the interesting engineering is in what happens next. Most fixed-priority kernels provide no notion of a deadline at all: a task has a period because the application delays itself by that amount, and a deadline that passes produces no event, no flag, and no counter. The system continues with a task that is late, taking longer than it should, and the failure appears downstream in a corrupted control output or a lost packet.

Making a miss observable

The first requirement is instrumentation. A task that is supposed to complete work within a period can compute how long its last iteration took and compare it against a configured limit; a task that must start by a certain time can compare the current tick against an expected release tick. Neither check is free, but both are a handful of instructions and a comparison, and they convert an invisible failure into a counter and a flag.

The pattern that costs the least and detects the most is a per-task deadline field and a single check at the task’s own iteration boundary. The check reads the tick at entry and at exit, and if the entry tick is later than the previous exit tick plus the period plus a tolerance, the task records a deadline overrun. The tolerance absorbs tick jitter and the analysis’s own slack.

Detection by measurement has a blind spot: a task that is late because it never runs at all cannot perform its own check. A watchdog on the task’s activity is what covers that case. The task updates a counter or a bit in a shared word on every iteration; a monitor — a higher-priority task or a hardware watchdog serviced by the monitor — checks that the counter advanced within the expected interval. A task that has hung, is blocked on a resource that will never be released, or has been starved by a priority bug leaves the counter stale, and the monitor sees it.

The counter must not be a simple flag. A flag set by the task and cleared by the monitor detects a task that never runs but not a task that runs and then hangs inside its own body; a counter that must advance by at least one between checks detects both. A bit per task in a shared word lets the monitor check all tasks with one read, and each task updates its own bit.

static volatile uint32_t liveness;     /* one bit per monitored task */

#define ALIVE(i)  do {                                     \
        taskENTER_CRITICAL();                              \
        liveness ^= (1u << (i));   /* toggle, not set */   \
        taskEXIT_CRITICAL();                               \
    } while (0)

/* monitor task, runs at the highest priority, one check per period */
static void monitor_task(void *arg)
{
    uint32_t previous = liveness;
    for (;;) {
        vTaskDelay(MONITOR_PERIOD);
        uint32_t now = liveness;
        uint32_t stale = previous ^ now;      /* bit unchanged since last */
        if (stale & watched_mask)
            report_fault(stale & watched_mask);
        previous = now;
    }
}

Toggling rather than setting is the detail that makes the check work without the monitor having to clear anything: each task flips its own bit on every iteration, so a bit that is unchanged since the previous check means the task has not completed an iteration in that interval. With a set-to-one flag the monitor would have to clear the bit after each check, and that clear races with the next set unless both are inside the same critical section. The cost of the toggle is that a task which completes an even number of iterations between checks cancels its own two flips, so the monitor’s period has to be shorter than the shortest legitimate interval between two iterations of any watched task.

The critical section around the toggle is needed because the word is shared with the monitor and with other tasks. A per-task byte or word avoids the shared word and the mutual exclusion, at the cost of more memory.

What recovery is allowed to do

After a miss, three classes of response exist, and they differ in what they preserve.

The first is to log and continue. The task completes its late iteration, the overrun counter increments, and the system continues with the assumption that the next iteration will be on time. This is appropriate when the miss is a transient caused by a one-off event and the downstream consumer tolerates the delay. It is not appropriate for a control loop where a late output is worse than no output, because the late computation is applied to a state that has moved on.

The second is to skip the iteration. The task resynchronises to the next period boundary, discarding the late work. The output has a gap of one period, and the counter records it. For sampling loops that integrate or filter, a skipped sample is a known input to the filter’s error budget; for a control loop, a skipped sample means the controller runs on an older measurement, and the control law’s stability margin has to account for it.

The third is to restart the task or the subsystem. Restarting a task means releasing its resources, running its initialisation again, and re-entering its loop. The difficulty is the state the task shares with others: a half-written buffer, a lock still held, a queue with items the task will read twice. A task that is restarted while holding a mutex leaves the mutex locked forever unless the kernel’s restart path releases it, and most kernels do not restart tasks at all.

The choice among these is a design decision that has to be made before the first miss, because the code that implements it sits on the fault path and must be simple enough to be trusted. The safest general pattern is to make the recovery action a state change that a supervisory task performs, so that the faulting task’s own code does not run during its recovery.

The hardware watchdog

A watchdog is a counter that resets the part if software does not refresh it. Its value is that it works when nothing else does: a hung task, a corrupted stack, an interrupt storm, a clock failure that the software cannot detect. Its cost is that a reset is a blunt instrument, and a watchdog that fires when it should not is worse than none, because it turns a recoverable glitch into a boot cycle.

The refresh policy determines whether the watchdog is useful. Refreshing it from a timer interrupt or from the idle loop proves only that the timer runs or that no task is busy, not that the application is healthy. Refreshing it from a monitor task that has checked the other tasks’ liveness proves considerably more, and the check is the reason the monitor’s own liveness must be proved by a different mechanism: the monitor is the one component whose hang would otherwise be undetectable.

The timing of the refresh is about the worst case, not the typical case. The window must be longer than the longest legitimate interval between refreshes, which includes the longest flash write, the longest blocking wait, and the longest burst of higher-priority work. A system that refreshes every 10 milliseconds with a window of 50 milliseconds is safe if nothing takes longer than 50 milliseconds; the flash write that takes 200 milliseconds is a designed-in false reset until the write is moved to a context that refreshes.

Parts that implement a windowed watchdog require the refresh to fall inside a range, which detects both a hung program and a program that is running too fast to be doing real work. The upper bound catches a refresh loop that fires without any real work in between, which a plain watchdog cannot see.

Reset is not always the safe response

The reflex is to let the watchdog reset the processor, but a reset is only safe if the system can be brought to a safe state from a cold boot, and if the reset itself does not cause a worse outcome. A motor controller that resets mid-motion drops its PWM outputs according to the pin’s reset state, which may or may not be the de-energised state; a relay that resets to closed is a hazard. The bypass circuitry and the pin configuration have to be chosen so that the reset state is the safe one, and that is a hardware decision that the fault-handling design cannot override.

The second consideration is time. A reset takes the boot time to produce a working system, which for a part with a slow flash and a long initialisation is tens to hundreds of milliseconds. A system with a deadline of a few milliseconds cannot meet it after a reset, so the reset path must be the last resort and a faster recovery must exist for the failures that can be handled in software.

The third is diagnosis. A reset that loses all evidence of the fault cannot be debugged. Storing a fault record — the faulting task, the program counter, the overrun counters, the watchdog’s own status register — to non-volatile memory before the reset is the difference between a field return that can be understood and one that cannot. The write has to happen before the watchdog fires, which means the monitor’s fault path has to complete within the watchdog window, or the record has to be written as part of the normal operation rather than on the fault path.

The safety standards treat all of this as a single requirement rather than a set of features: a system must detect its own failures and reach a safe state within a bounded time. The watchdog is one of the detection mechanisms, the liveness counters are another, and the timing requirements on both are derived from the same hazard analysis. Interrupt latency and kernel overhead supplies the numbers for the detect-and-react path, and stack and memory budgeting covers the guard that turns an overflow into a fault rather than a corruption.

What the schedule owes the fault path

A fault handler that runs at a priority above the monitor competes with the monitor for the processor, and a fault that repeats can starve it. The usual design is to have the monitor record the fault and take the safe action directly, rather than to have it call back into the faulting subsystem.

A supervisor task that must handle the fault also needs a priority above every application task, because a fault in a low-priority task must be handled even while a high-priority task runs. That priority puts the supervisor in the response-time analysis as a term on everyone’s interference, and its own response time has to be checked with the same recurrence as any other task. A supervisor that is analysed as a soft task and acts as a hard one is a gap in the argument that no amount of testing closes.