What Occupancy Actually Counts

Occupancy is usually reported as a percentage, which invites the reading that it measures how busy the machine is. It does not. It measures how many warps are resident on a streaming multiprocessor relative to the maximum that SM can hold, at a moment when those warps may all be stalled. A kernel at 100 percent occupancy can be idle; a kernel at 25 percent occupancy can be saturating the datapath. The number is a capacity statement, and its usefulness depends entirely on what it is compared against: the latency the kernel must hide.

Getting this right requires knowing what the cap is made of. Three resources limit resident warps independently: registers per thread, shared memory per block, and hard limits on warps, blocks, and threads per SM. Whichever binds first sets the occupancy, and the binding resource is frequently not the one the programmer was watching.

The ratio and its denominator

An SM can hold a fixed number of resident warps, a fixed number of resident blocks, and a fixed number of resident threads. The maximum warps per SM follows from the thread cap divided by the warp width, since warps are the scheduling unit. Across the architectures in common use, the per-SM thread cap has ranged from 1024 to 2048 threads, giving 32 to 64 warps per SM; the block cap is typically 16 or 32 resident blocks. These are architectural constants, published in the compute capability tables of the programming guide, and they differ by architecture rather than by kernel.

Occupancy for a given kernel on a given SM is:

warps_resident = min( warps_per_SM_limit,
                      blocks_per_SM_limit * warps_per_block,
                      floor( regs_per_SM / (regs_per_thread * 32)),
                      floor( smem_per_SM / smem_per_block) * warps_per_block )
occupancy      = warps_resident / warps_per_SM_limit

Each term is a floor, and that is the first thing to notice: allocation is quantized. Registers are allocated per warp in granular units, so the cost is not proportional to the count. A kernel using 33 registers per thread can consume 40/32 of what a 32-register kernel does, because the per-warp allocation is rounded up to a unit before the division. The same quantization applies to shared memory, which is allocated per block in fixed-size units, so a block requesting slightly more than a unit pays for the whole unit.

The second thing to notice is the multiplication by warps_per_block in the block and shared memory terms. A block’s shared memory allocation caps how many blocks fit, and each block brings its own warps. A kernel with large blocks and modest shared memory per block can therefore be limited by shared memory even though the per-thread footprint is small, because the granularity of allocation is the block.

Registers are the usual ceiling

Register pressure binds more often than anything else, because register count is a per-thread property that the compiler sets and the programmer rarely inspects. If an SM has 65536 registers and a warp needs 32 registers per thread, one warp consumes 1024 registers, so 64 warps fit. At 40 registers per thread a warp consumes 1280, and 51 warps fit, since the register file is not partitioned into fractional warps. At 64 registers per thread, 32 warps fit, half the maximum. At 128 registers per thread, 16 warps.

The mechanism behind this is a hard partition. The register file is physically banked, and a warp’s registers are assigned to fixed slots at launch; they cannot be reclaimed while the warp is resident and cannot be shared between warps. So the count is a static reservation, made once, for the lifetime of the warp. A kernel that needs many registers to avoid spills and a kernel that needs many warps to hide latency are asking for the same physical resource in incompatible ways. The compiler’s decision about how aggressively to unroll loops and keep values live directly sets the occupancy ceiling, and this trade is the subject of register allocation and the occupancy ceiling.

The classic failure mode is the compiler choosing a register count just above an allocation boundary. A kernel at 32 registers per thread may fit 64 warps; at 33 the rounded allocation may fit only 51. The performance difference between those two kernels can exceed the difference between a well-written and a poorly-written inner loop, and it appears without any change to the source’s arithmetic. Setting a per-kernel launch bound on register use forces the compiler to fit the budget, at the cost of spills to local memory, which are themselves memory traffic that the extra warps were supposed to hide.

Shared memory and the block granularity

Shared memory is allocated per block from a fixed per-SM pool, tens of kilobytes on older parts and a couple of hundred kilobytes on recent ones, with the pool size and the maximum per-block allocation both architecture-dependent. Because the allocation unit is the block, a kernel with a large per-block allocation reduces the number of resident blocks, and the number of resident warps falls with it even if each block is large.

Two effects follow. First, a block that allocates more shared memory than it uses wastes occupancy; the allocation is a declaration, not a demand. Second, the product of resident blocks and warps per block can exceed the warp cap, in which case the warp cap binds and the extra shared memory buys nothing. The useful mental model is that shared memory limits blocks, blocks supply warps, and the warp cap truncates the result.

On several architectures the shared memory pool is shared with a fixed-function cache, and the split is set per kernel rather than fixed by the architecture alone, so the pool a kernel sees is its carveout rather than the whole combined array. The partition belongs to the memory hierarchy rather than the scheduling model; what matters here is that it changes the denominator of the shared memory term, and that a kernel can be limited by a pool smaller than the one the architecture reports.

Why more resident warps help, and when they stop

The point of resident warps is latency hiding by interleaving. An SM issues one instruction per warp scheduler per cycle; if the selected warp’s next instruction depends on a load that will take hundreds of cycles, the scheduler must issue something else or issue nothing. With more resident warps there is more likely to be a ready warp, so the datapath stays occupied.

The arithmetic is simple enough to state without measurement. If a warp’s instruction stream stalls for L cycles every k instructions, a single warp can keep a scheduler busy only if k ≥ L. With W warps per scheduler, the scheduler is saturated when W·k ≥ L, roughly speaking, because the warps’ stalls interleave. So the occupancy needed is proportional to the latency-to-work ratio of the kernel, not to a fixed target.

That framing explains the cases where low occupancy is fine. A kernel that is compute-bound with long runs of independent arithmetic between memory accesses has a small L/k ratio and saturates at low occupancy. A kernel dominated by dependent loads from global memory has a large ratio and needs many warps. A kernel with heavy shared memory use and explicit synchronization needs enough warps per block to fill the block’s work, and enough blocks that the SM stays full; adding warps past that point does nothing because the barrier serializes them anyway.

It also explains why 100 percent occupancy is not a goal. Every warp costs registers and scheduler slots. If the kernel saturates at 50 percent, the other 50 percent is capacity that could be spent on more registers per thread, which might remove spills and shorten the instruction stream. The trade is real in both directions, and the honest way to state it is that occupancy is a knob whose correct setting is determined by the latency the kernel must cover, which is a property of the memory access pattern.

The measurement trap

Occupancy reported by profiling tools is usually theoretical: the value computed from the resource limits for the launch configuration. Achieved occupancy is a different number, measured as the average fraction of warp slots actually occupied over the kernel’s life. The gap between them is the tail effect.

A grid whose blocks do not divide evenly into the SM capacity leaves the last wave partially filled. If a kernel runs four waves of blocks, the final wave may occupy a quarter of the slots, and the achieved occupancy averages to something below theoretical even though every full wave was at the cap. This is a property of grid size and block count, not of the kernel’s resource use, and it is why small grids with large blocks underperform their occupancy numbers.

A second gap comes from the block scheduling order. Blocks are assigned to SMs as slots free up, so a kernel with variable per-block runtime (a data-dependent loop count, for instance) can leave some SMs idle at the end while others still work. Neither effect is visible in the theoretical figure.

The practical rule that follows: compute the theoretical occupancy to find which resource binds, then compare it against the latency-hiding requirement derived from the access pattern, and treat the achieved figure as the only evidence about what the machine did. Warps that are resident but stalled are counted as occupancy and contribute nothing, which is why the model in latency hiding and the arithmetic of warp counts is the necessary companion to this one.