DevOps · On-Call

Rotation Coverage Math and the Handoff Contract

How 168 hours of weekly coverage divide across a team, what shift length costs, and what has to cross the boundary at every handoff.

A week contains 168 hours, and every on-call schedule is an attempt to cover some subset of them with a finite number of people. That sentence contains the whole design problem. The arithmetic is trivial and unforgiving: coverage that looks generous on a calendar turns out to require far more person-hours than a team has, and the shortfall is absorbed either by overworked responders or by hours nobody is actually watching.

The arithmetic of 168 hours

One layer of continuous coverage consumes 168 person-hours per week. Spread across a team of six, that is 28 hours per person per week on top of normal work, which is most of a working week before any incident is handled. Spread across five, it exceeds a standard working week. The number does not improve with better tooling; it is a property of the clock.

Google’s SRE book recommends capping on-call at roughly a quarter of an engineer’s time, which for a forty-hour week is ten hours. Covering 168 hours at ten hours per person requires about seventeen engineers for a single layer. That figure explains a great deal of observed practice: small teams do not run continuous coverage, they run business-hours coverage with a best-effort night path, or they accept a much higher fraction of time on call than the guideline suggests.

Two layers, primary and secondary, double the requirement. The secondary exists so that an unacknowledged page has somewhere to go, and so that the primary can sleep through a night in which nothing happens without being the only line of defense. The cost is real and should be stated in the same arithmetic: adding a secondary to a six-person rotation takes each person from 28 to 56 hours per week of coverage, which is not sustainable, so small teams typically share a secondary across services or leave the escalation path to a manager.

Coverage can also be defined by hours rather than by layers. Eight hours a day on weekdays is forty hours a week and fits one person with no overtime, but it leaves 128 hours uncovered. Those hours are not free: the pager still fires, and the response is either an informal best-effort arrangement or nothing. Making the boundary explicit, so that a night-time page goes to a documented fallback rather than to whoever happens to look at the channel, is more honest than pretending the schedule covers it.

Shift length trades fatigue against handoffs

Once coverage is continuous, the next decision is how to divide the day. Three eight-hour shifts give a clean division and put each responder on duty for a third of the day. Two twelve-hour shifts give longer blocks and fewer transitions. A single twenty-four-hour shift minimises transitions entirely.

The trade is between two costs. Longer shifts reduce the number of handoffs, and every handoff costs context. Shorter shifts reduce sustained sleep disruption and the performance decrement that comes with being awake through a night, at the price of more transitions, more chances for state to be lost, and more people who need to be briefed on a system they have not looked at since their last shift.

Handoff cost is not zero. A useful transfer of an active incident takes on the order of fifteen to thirty minutes: reading the timeline, understanding the current hypothesis, checking what has already been tried, and confirming what is in flight. A schedule with twenty-one handoffs per week at twenty minutes each spends seven hours a week on transfer alone, before any incident work. That number is the argument for twelve-hour shifts in many organizations, and the counter-argument is the documented degradation of alertness across a night that long. There is no configuration that avoids choosing.

The number of handoffs an incident experiences is its duration divided by the shift length, rounded up. A four-hour incident during an eight-hour rotation usually crosses no boundary; a twelve-hour incident crosses at least one. The practical consequence is that long incidents accumulate transfer risk, and the case for keeping one owner across a boundary, with the incoming person acting as support rather than as a replacement, gets stronger as the incident gets longer.

Where the schedule and the calendar disagree

Schedules are almost always expressed as recurrences, and the recurrence has to be anchored to a time zone. RFC 5545 defines the recurrence rule syntax that most calendar and scheduling systems borrow: a weekly rule with a day list, plus exception dates for overrides. The anchor choice determines what happens at a daylight-saving transition.

If the shift is defined in local wall-clock time, a daily 09:00 handoff occurs 23 hours after the previous one in spring and 25 hours after it in autumn. The people involved experience the handoff at the same local time, but the coverage interval changes length, and any computation of hours worked is wrong twice a year unless it uses elapsed time rather than wall-clock subtraction. If the shift is defined in UTC, the interval is always exactly 24 hours, but the local start time drifts by an hour, so a shift that was 22:00 to 06:00 becomes 21:00 to 05:00 for part of the year. Neither representation is wrong; the error is mixing them, for example computing payroll in local time while the pager resolves targets in UTC.

Overrides are the second disagreement. A swap between two engineers is an exception to the recurrence, and an exception that exists only in a chat message is not part of the schedule. The system that pages people reads the schedule, so a swap that is not recorded there produces a page to someone who believes they are off, or a page to nobody. The same applies to holiday coverage and to temporary reductions in team size: an override with an end date is fine, an override without one becomes a permanent silent change.

What has to cross the boundary

The handoff is a transfer of state, and the state is larger than the list of open incidents. A written handoff note in a fixed location, posted before the outgoing responder goes offline, is the minimal mechanism. Read-back, in which the incoming responder restates the open items, catches the cases where the note assumed context the reader does not have.

The contents that matter most are the ones that are not visible from a dashboard. Open incidents and their ownership are visible. The following are not:

  • Silences and their expiry times, because a silence that expires at 03:00 is a page waiting to happen.
  • Known-degraded components that are not currently alerting, such as a replica that is out of the pool or a queue that is draining slowly.
  • Changes in flight: deploys, migrations, configuration rollouts, and any freeze that is in effect.
  • Scheduled maintenance and its expected blast radius.
  • Expiring things: certificates, credentials, and quota that will run out within the shift.
  • The escalation contacts for the day, including anyone who is traveling or unreachable.

The last item is the one most often omitted and the one that hurts most, because the escalation policy resolves to a rotation, not to a person’s current availability. A handoff that says “the database team’s secondary is at a conference” is worth more than a page of incident summaries.

Failure modes that recur

An unfilled slot is the obvious one. It happens when the schedule is built from a template and a new team member is added to the roster but not to the rotation, or when someone leaves and their shifts remain. A validation check that resolves every future shift to at least one person, run on a schedule and alerting on gaps, catches it before the gap becomes an incident with no responder.

The shadow schedule is subtler. A team keeps the real arrangement in a spreadsheet or a pinned message while the paging system holds a stale approximation. The two diverge slowly, and the divergence is discovered when a page reaches the wrong person at the wrong time. The only durable fix is to treat the paging system’s schedule as the single source of truth and make any other representation derived from it.

The third is handoff debt. When the outgoing responder is tired, the note gets shorter, and the incoming responder starts the shift with less context. Over a run of shifts the notes decay toward a list of incident numbers with no state. A fixed template, even a short one, resists the decay better than an instruction to write a good summary.

Coverage that spans multiple regions adds a further dimension, because the handoff is then also a change of working hours and of local context; that case is treated in follow-the-sun coverage. The mechanics of what happens when a page goes unanswered remain governed by the timers described in escalation policies as state machines, and the schedule only determines who those timers reach.

References

  1. Site Reliability Engineering, Chapter 11: Being On-Call — Google
  2. RFC 5545: Internet Calendaring and Scheduling Core Object Specification — IETF
  3. Time Zone Database — IANA