POSIX Real-Time Priority Classes and the Bandwidth Cap

The general-purpose policies in this cluster all answer “how should the CPU be divided”. The real-time policies answer a different question: “which thread must run now, regardless of how the division has gone so far”. POSIX defines two such policies, SCHED_FIFO and SCHED_RR, alongside the ordinary SCHED_OTHER, and the semantics of the two are short enough to state in a paragraph. Almost all the engineering around them is in the constraints that make them safe to expose.

What FIFO and round-robin actually promise

SCHED_FIFO is strict priority with no time slicing. A thread at the highest priority runs until it blocks, until it calls sched_yield, or until a thread with a strictly higher priority becomes runnable. A thread at the same priority does not preempt it; it waits until the running thread gives up the CPU. Among threads at equal priority, the order is first-in-first-out: a thread that becomes runnable goes to the tail of its priority’s queue, and the head runs next.

SCHED_RR is the same policy with a time quantum added. When a thread exhausts its quantum it goes to the tail of its priority’s queue and the next thread at that priority runs. Preemption by a higher priority and the FIFO ordering within a priority are unchanged. The quantum is implementation-defined; POSIX does not fix a value, only that it exists and applies to the round-robin policy.

Both policies are defined only in terms of relative priority. POSIX requires that an implementation provide at least 32 distinct priority levels for each policy, and that sched_get_priority_max and sched_get_priority_min report the range. The numbers themselves are not portable: a program that hard-codes 99 as the maximum is relying on a specific implementation, and the same code on another system may find its requested priority out of range and receive EINVAL.

The important structural property is that these policies have no notion of fairness. A SCHED_FIFO thread that never blocks and never yields at a priority above everything else will run forever. There is no quantum to expire, no vruntime to catch up, and no balance to restore. That is the point, since a control loop that must sample every millisecond cannot tolerate being descheduled for a slice, and it is also the hazard.

Why the privilege boundary exists

Because a runaway SCHED_FIFO thread can make a system unusable, the ability to set a real-time policy is restricted. POSIX gives the mechanism but leaves the authorisation to the implementation; the common model is a resource limit that names the maximum real-time priority an unprivileged process may use, with zero meaning none. Raising the limit requires privilege.

The consequence for library and application design is that a request for a real-time policy can fail at run time even though the call is correct, and the failure is not an error in the program. Code that treats a failed sched_setscheduler as fatal will refuse to start on a system where it would have run acceptably at ordinary priority. Code that ignores the failure will silently run at the wrong priority and miss deadlines without any indication of why. The defensible behaviour is to attempt the change, check the result, and fall back to a documented degraded mode.

Priority selection has a second constraint that is easy to get wrong. Real-time priorities are absolute, not relative to the process. A thread at priority 50 outranks every thread in every process at priority below 50, including kernel threads that perform work on the system’s behalf. Assigning high priorities liberally therefore does not just affect the assigning process; it can starve unrelated subsystems. The usual discipline is to reserve the top of the range for a small number of threads with genuinely hard deadlines and to place everything else as low as the deadline permits.

#include <pthread.h>
#include <sched.h>
#include <stdio.h>

/* Set FIFO at the lowest real-time priority; report why it failed. */
static int make_realtime(int prio_min)
{
    struct sched_param sp = { .sched_priority = prio_min };
    int rc = pthread_setschedparam(pthread_self(), SCHED_FIFO, &sp);
    if (rc == EPERM)
        fprintf(stderr, "no real-time privilege; continuing at SCHED_OTHER\n");
    else if (rc == EINVAL)
        fprintf(stderr, "priority %d outside this system's range\n", prio_min);
    return rc;
}

EPERM and EINVAL are distinct failures with distinct remedies: the first is authorisation, the second is a bad argument. Conflating them produces diagnostics that send the reader to the wrong place.

The runtime cap

Even with the privilege boundary in place, a privileged process can hang the machine. The standard defence is a cap on how much of each period the real-time policies may consume in aggregate. The kernel maintains a period and a runtime; when the accumulated real-time execution within the current period reaches the runtime, the real-time threads are throttled until the next period begins, at which point ordinary threads run.

The common implementation expresses the cap as two numbers, a runtime budget and a period over which it may be spent, with the runtime set to a fraction of the period so that ordinary threads are never completely shut out. On Linux the system-wide settings are sched_rt_period_us and sched_rt_runtime_us, exposed under /proc/sys/kernel/; with group scheduling enabled the same numbers become the global budget against which per-cgroup files named cpu.rt_period_us and cpu.rt_runtime_us are charged. When the runtime value is set to -1 the cap is disabled and a real-time thread can run without limit. The headroom is not a fairness gesture; it is what keeps the machine responsive enough to diagnose and kill the offending process. A SCHED_FIFO thread that spins without blocking will, under this scheme, still leave a sliver of CPU for everything else. The exact default values are a policy choice and vary by system, so they belong in the configuration rather than in a program’s assumptions about how much it may consume.

Throttling changes the semantics that a real-time program can rely on, and this is the part that surprises people. A thread that believes it has absolute priority will, once the cap is reached, be descheduled for the remainder of the period, which may be long in comparison to the thread’s own timing. A control loop designed around a one millisecond period will miss many deadlines in a row, and it will do so in exactly the situation where it is working hardest. The cap is therefore not a transparent safety net; it is a change in the policy’s contract, and a system that needs a hard guarantee must either raise the cap deliberately (accepting the risk) or use a mechanism that accounts for the budget explicitly.

That second option is the reservation-based approach: instead of claiming unbounded priority and being throttled arbitrarily, the thread declares a budget and a period, and the scheduler enforces the budget as part of admission rather than as a punishment. The kernel’s deadline policy and the constant bandwidth server that underlies it are described in Deadline Servers and Admission Control.

Mixing policies on one machine

A machine running real-time and ordinary threads has to decide how the two interact, and the answer is not symmetric. Real-time threads preempt ordinary ones whenever they become runnable; ordinary threads run only when no real-time thread wants the CPU and the cap has not been exhausted. This is a strict hierarchy, and it means the response time of an ordinary thread is a function of the real-time load, not just of its own priority.

The practical consequence is that real-time load must be bounded before it is deployed, not after. A single real-time thread that wakes every millisecond and runs for half a millisecond consumes half the CPU of one core and, on a machine with one core, halves the throughput of everything else. If several such threads exist on the same core, the sum of their utilisation is the fraction of the CPU they take from ordinary work, and once that sum approaches one, ordinary work stops making progress for long stretches.

This is the same arithmetic that schedulability analysis performs for a fixed-priority real-time task set, applied informally. The formal version, the utilisation bound that determines in advance whether a set of periodic tasks can meet all deadlines, is the subject of EDF, Utilization Bounds, and Schedulability Tests.

Where the interaction becomes genuinely difficult is when a real-time thread must acquire a lock also taken by ordinary threads. The real-time thread’s priority is absolute, but the lock holder’s is not, so the real-time thread can block behind an ordinary thread that is itself being preempted by other ordinary threads. The real-time priority does not help, because the dependency runs through the lock rather than through the scheduler. Inheritance addresses the case where the holder’s priority is meaningful; a real-time thread blocking on a mutex held by a SCHED_OTHER thread is a design error more than a scheduling one, and the mechanisms that mitigate it are described in Priority Inversion and Inheritance.

What the policies do not give

Neither SCHED_FIFO nor SCHED_RR provides a deadline. Priority is a static rank, and meeting a deadline by ranking depends on the deadline structure being expressible as a total order, which it is only when all threads have the same period and deadlines equal to periods. When periods differ, a higher-priority thread with a long period can be the wrong choice at a given instant, and no static assignment fixes that in general.

Nor do they provide any admission control. Setting a thread to SCHED_FIFO succeeds regardless of whether the resulting load can meet its deadlines. The system will accept a task set that is unschedulable and then miss deadlines, which is the worst possible behaviour for a system with a correctness requirement: it fails late, under load, and without a clear signal that the configuration was infeasible from the start.

Nor do they account for the cost of the mechanism itself. Every preemption by a real-time thread pays the switch cost described in Run Queues and the Cost of a Context Switch, and a real-time thread that wakes at high frequency forces a preemption each time. At a wakeup rate in the tens of thousands per second, the switch overhead becomes a significant fraction of the CPU, and the real-time thread’s own execution is no longer the dominant term in its response time. The usable wakeup rate is bounded by the switch cost, not by the priority.

The gap between “runs first” and “finishes in time” is what the deadline-based mechanisms fill. They require the program to state a period and a budget rather than a rank, and they require the system to refuse a request that cannot be satisfied, a stricter contract that is more useful precisely because it can fail at configuration time instead of at run time.