Escalation Policies as Timeout-Driven State Machines

An escalation policy is usually drawn as a list of names: first the primary, then the secondary, then the engineering manager. That drawing hides the part that matters. What actually runs is a state machine with one clock per step, and the behaviour of the system under load is decided almost entirely by how those clocks are set and what event stops them.

A step is a set of targets bound to a deadline

Each step in a policy has two independent parameters: a set of targets and a timeout. The targets are typically a rotation rather than a named individual, so the policy survives a roster change without editing. The timeout is the amount of time the system waits for an acknowledgement before it stops treating the step as active.

A third parameter, the notification repeat interval, controls retries inside a step. A step with a ten-minute timeout and a three-minute repeat interval will attempt contact up to four times before advancing. These two numbers are frequently conflated, and conflating them produces two different failures: a repeat interval longer than the timeout means the step advances before the first retry, so a person who was away from the phone for two minutes never hears about the incident at all. A repeat interval of thirty seconds against a paged phone produces a burst that looks, from the receiving end, like a malfunction.

The rotation indirection is worth stating explicitly because it is the source of a whole class of routing bugs. The policy holds a reference to a schedule; the schedule resolves to a person at the moment of notification. If the schedule is misconfigured, or an override was applied for last week and never removed, the policy is structurally correct and operationally useless. Validation therefore has to happen at resolution time, not at policy-edit time.

The clock starts at notification, not at detection

The timeout on step one does not begin when the metric crossed its threshold. Between the crossing and the first ring there are several distinct delays: the scrape or evaluation interval of the monitoring system, the pending duration that a rule requires before it fires, the grouping wait imposed by the alert router, and the transport delay of the paging provider. Each is small; together they routinely account for minutes.

This matters when the timeout is chosen to meet a response target. A service with a five-minute service-level objective for human acknowledgement cannot be protected by a five-minute step timeout, because the measurement starts earlier than the timer does. The step timeout has to be budgeted against the residual, and the residual is only visible if the monitoring pipeline’s own latencies are known. Rules that wait several minutes in a pending state before firing are common and are not free.

The clock is also not the same clock as the responder’s. Deadlines should be stored as absolute instants in a monotonic or UTC frame, not as durations that get decremented by a scheduler tick. A process restart that reloads a duration restarts the wait; a process restart that reloads a deadline does not. Systems that store “time remaining” rather than “time at which to act” extend every incident by however long the restart took, and do so silently.

Acknowledgement stops escalation, not the incident

The acknowledgement event is the only thing that halts the chain. When any target in the active step acknowledges, the system cancels the remaining timers for that incident and stops notifying further steps. Everything downstream of that decision depends on a distinction that is easy to lose: an acknowledged incident is not a resolved incident.

Resolution is a separate event, produced either by the monitoring system observing that the condition has cleared or by a human declaring the incident over. A policy that treats acknowledgement as the end of the lifecycle will keep escalating nothing while the service continues to fail, and the dashboards will show a green acknowledgement where an operator expects to see an open problem. The two states need to be visible separately: acknowledged and unresolved is the state in which most real work happens, and it is the state most often missing from summary views.

There is a race at the other end as well. If the underlying alert clears before anyone acknowledges, the incident closes on its own. The responder may then receive a notification and a resolution within seconds of each other, which is worse than receiving neither, because it trains people to distrust the pager. Short-lived alerts are better handled by a pending duration before the first notification than by a fast auto-resolve after it.

Simultaneous targets and the cost of waking people

A step with several targets can notify all of them at once or one after another. Notifying everyone at once minimises the time to the first human response and maximises the number of people woken for an incident that one of them could have handled. Serial notification inside a step inverts both: each target gets a full timeout before the next is tried, so the step’s effective latency is the timeout multiplied by the number of targets.

The usual compromise is a small simultaneous first step, because the cost of a slow first response is paid by users, and a serial second step, because by then the incident is already late and the marginal value of speed is lower than the marginal cost of waking a third person. That structure is a judgment about the service, not a universal rule, and it should be revisited when the composition of the rotation changes.

Where the state machine breaks

The first failure is acknowledgement used as a snooze. A responder who is driving, or already inside another incident, taps acknowledge to silence the page and intends to look later. The policy cannot tell the difference between that and genuine ownership, so escalation stops and the incident waits. Mitigations are procedural rather than structural: a separate “snooze” action with an explicit short expiry, or a rule that an acknowledged incident with no activity after some interval re-notifies the same person rather than escalating.

The second is the empty final step. Policies are often built by appending steps over time, and the last one ends up pointing at a rotation that no longer has members, or at a manager who left. Nothing validates that the terminal step can reach a human. A simple check at configuration time, resolving every step’s targets and rejecting the policy if any resolves to an empty set, eliminates a failure mode that otherwise surfaces only during the incident that needed it most.

The third is collision. Escalation policies are per-incident, and they have no knowledge of what else the target is doing. Two simultaneous incidents will page the same secondary, and the second page arrives while the first is being worked. The policy is behaving correctly; the system as a whole is not. Detecting that a target is already engaged requires state that lives outside the policy, which is one of the reasons incident roles and explicit ownership exist as separate mechanisms.

# Deadlines rather than durations, so a restart does not restart the wait.
from dataclasses import dataclass

@dataclass
class Incident:
    step: int = 0
    deadline: float | None = None      # monotonic seconds
    acknowledged_by: str | None = None

def tick(inc: Incident, policy, now: float, notify) -> None:
    if inc.acknowledged_by is not None:
        return
    if inc.deadline is None:
        notify(policy[inc.step].targets)
        inc.deadline = now + policy[inc.step].timeout
        return
    if now >= inc.deadline:
        if inc.step + 1 == len(policy):
            return                      # terminal step: nothing left to try
        inc.step += 1
        notify(policy[inc.step].targets)
        inc.deadline = now + policy[inc.step].timeout

The sketch omits the repeat interval deliberately. Retries inside a step are a separate timer per step, and collapsing them into the escalation deadline is exactly the conflation described above.

Sizing the timeouts against the response target

The total time before the last step is reached is the sum of the preceding step timeouts plus the transport delays, and that sum is the number that must fit inside the service’s tolerance for undetected failure. A four-step policy with five-minute steps reaches its fourth target twenty minutes after the first notification, which is a long time for a hard dependency to be down.

Two constraints bound the steps from below. A timeout shorter than a realistic pickup time, which includes finding the phone and deciding whether to answer, produces escalation to a second person while the first is already responding. And a timeout shorter than the monitoring pipeline’s own detection delay is meaningless, because the second step fires before the first has any information to act on. The practical consequence is that fast escalation and low noise pull in opposite directions, and the resolution is usually fewer steps with better targets rather than more steps with shorter timers.

The severity of the incident should also feed the policy, since a routing decision that ignores severity either over-pages for trivial conditions or under-pages for serious ones. How severity is defined and how it selects a channel is covered in severity classification as a routing input. What the policy cannot do is decide whether the page was worth sending in the first place; that judgment belongs to the alerting rules and to the grouping logic that decides how many pages one failure produces.