Time-Slicing and What a GPU Context Switch Costs

Space-slicing gives each tenant a fixed share and no interference; it also gives each tenant a share that cannot change. Time-slicing gives the opposite: one tenant at a time has the whole device, and the operator can decide how much time each gets. The two designs trade an idle resource for a switching cost, and the switching cost on a GPU is unlike anything on a CPU.

A CPU context switch saves a few dozen registers and a stack pointer, and modern hardware supports several register files so that a switch can be a pointer change rather than a copy. A GPU must save the state of every thread it has in flight: for a device with tens of thousands of resident threads, each with its own register allocation, plus the shared memory contents of every resident block, plus the contents of the L1 and L2 caches if the incoming context is not to start cold. Saving all of that is a transfer of megabytes, and it is why GPU preemption is usually described in terms of what can be avoided rather than what can be saved.

Three granularities of preemption

The useful distinction is not “preemptible or not” but “at what point can the hardware be interrupted”. Each level bounds a different quantity.

At the finest level, the hardware can stop scheduling new warps and let resident warps drain. No state is saved; the context stays resident, occupying registers and shared memory, and simply makes no progress. This is cheap to invoke and expensive to wait for, because the time to quiesce is the time for the longest-running warp to finish, which for a kernel with a long-running loop can be unbounded.

At the next level, the hardware saves the per-thread architectural state, meaning registers and program counters, while discarding shared memory. The context leaves the device’s execution resources, and the incoming context can be loaded. The cost is proportional to the number of resident threads and their register usage, and it must be followed by reloading the same amount for the incoming context. Any data in shared memory is lost, so a block that is mid-execution must either be restarted or must have been at a point where shared memory contents were disposable.

At the coarsest level, the hardware waits for the kernel to complete and then switches. Nothing is saved, and the switch cost is the tail latency of the running kernel. This is what happens when a device has no preemption support at all, and it is why a long-running kernel on such a device can monopolise it for as long as it runs.

The level a device supports is a hardware property, and the levels differ in whether the driver can choose them or the hardware picks. Where the hardware supports instruction-level preemption, the compute unit can be interrupted at a bounded instruction count, and the amount of state to save is whatever is architecturally live at that point. That is the strongest form and the most complex, because every instruction must be restartable and every architectural state element must be identifiable.

What the state actually consists of

A thread’s architectural state on a GPU is small per thread and enormous in aggregate. Registers are allocated per thread from a per-block pool, and a block with 256 threads using 64 registers each holds 16,384 registers, which is 64 KiB of register file. Shared memory is allocated per block, and a modern device offers on the order of a hundred kilobytes per compute unit, so a resident block may hold most of that. A device with a hundred compute units and eight resident blocks each is holding tens of megabytes of on-chip state at any moment.

The save path therefore writes to device memory, which means the switch consumes the bandwidth the tenants are competing for. This is the reason time-slicing is not free even when the switch itself is rare: a switch during a bandwidth-bound workload steals bandwidth from both the outgoing and the incoming tenant.

The L2 is the part that is usually not saved. A context switch that leaves the L2 contents in place means the incoming tenant starts with a cache full of the outgoing tenant’s lines, which it will evict. The eviction is not a correctness problem, since the L2 is not architecturally visible, but it is a performance problem that shows up as a cold-start penalty after every switch. Devices that partition the L2 by context tag avoid the pollution at the cost of reducing the effective capacity for each context.

The scheduling policy is where the trade is made

Given a preemption mechanism, the operator chooses a policy, and the policy determines whether oversubscription helps. The two useful policies are round-robin with a time quantum and priority with preemption.

Round-robin with a quantum has the property that the quantum must exceed the switch cost by enough that the switch is a small fraction of the slot. If a switch costs a millisecond and the quantum is a millisecond, the device spends half its time switching and the throughput is halved. Choosing a quantum large enough to amortise the switch makes the scheduling latency large, which matters for tenants with interactive work: a tenant waiting for its turn waits up to the sum of the other tenants’ quanta.

The arithmetic is worth writing down because it fixes the only two numbers a policy can control:

/* fraction of device time available to tenants, given a round-robin policy */
static double useful_fraction(double quantum_us, double switch_us)
{
    /* each turn costs quantum + switch, and the switch is not useful work */
    double cycle = quantum_us + switch_us;
    return quantum_us / cycle;
}

/* worst-case time a tenant waits for its next turn */
static double latency_us(unsigned tenants, double quantum_us, double switch_us)
{
    return (tenants - 1) * (quantum_us + switch_us);
}

Taking eight tenants, a 100 µs quantum, and a 200 µs switch as illustrative values, a third of the device time goes to useful work and the rest to switching, and the wait for a turn is 2.1 ms. Doubling the quantum to 1 ms raises useful time to about 83 percent and raises the wait to 8.4 ms. The direction of both curves is fixed, so the choice is a position on a line rather than a search.

Priority with preemption lets a high-priority tenant interrupt a low-priority one, which is useful when the high-priority work is latency-sensitive and the low-priority work is throughput-oriented. The cost is that the preempted tenant loses its cached state and, if shared memory is discarded, may lose progress on partially executed blocks. A tenant that is preempted repeatedly makes progress at a rate that depends on its neighbours, which is a form of interference that no isolation boundary prevents.

The third policy is not a policy at all: exclusive assignment for the lifetime of the tenant. It has no switch cost and no interference, and it has no oversubscription. That is the passthrough and virtual function case, and it is the right choice whenever the workloads are long enough that oversubscription would not help anyway.

Concurrency rather than preemption

There is a middle path that avoids most switching: let several contexts be resident at once and let the hardware interleave them. If two contexts’ blocks fit on the same compute unit simultaneously, the hardware’s warp scheduler can issue from either, filling each other’s stalls. No state is saved, because both contexts’ state is resident. The cost is that each context gets fewer registers and less shared memory, and the benefit is that the device stays busy during one context’s memory stalls.

This is how concurrent kernel execution works, and it is also the mechanism behind multi-tenant sharing on devices without preemption. The isolation it provides is weaker than partitioning, because the contexts share caches and memory bandwidth and their occupancy depends on each other’s resource demands. A tenant whose kernel uses the maximum registers per thread prevents another tenant’s kernel from being co-resident at all, which is a denial of service that the scheduler must be aware of.

Making concurrency predictable requires the scheduler to account for resource footprints, not just time. A scheduler that admits a context based on its register and shared-memory demand can guarantee that a previously admitted context continues to make progress; one that admits on a first-come basis cannot. This is the same admission-control problem that appears in memory oversubscription, where the resource being admitted is device memory rather than registers.

Measuring the switch without measuring the device

The switch cost is not directly observable from inside a tenant, which makes it easy to misattribute. A tenant that sees its kernel take twice as long under a co-tenant may be suffering from switch overhead, from cache pollution, or from bandwidth contention, and the three have different fixes.

The distinguishing signal is the relationship between the slowdown and the work size. Switch overhead is a fixed cost per switch, so it shows up as a constant additive term and dominates short kernels. Cache pollution is proportional to the working set, so it shows up when the tenant’s working set exceeds what survives a switch. Bandwidth contention is proportional to the tenant’s traffic, so it scales with the memory intensity of the kernel and is largely independent of its duration.

A scheduler that does not distinguish these will tune the wrong parameter. Increasing the quantum reduces switch overhead and worsens latency; partitioning the L2 by context reduces pollution and reduces capacity; prioritising bandwidth-sensitive tenants reduces contention and starves the rest. The performance reasoning that separates these effects is what makes a sharing policy defensible rather than arbitrary.