GPU Computing · GPU Performance
Occupancy as a Latency-Hiding Budget
How resident warps are rationed by registers, shared memory, and block granularity, and how to compute the number of concurrent requests a given occupancy can sustain.
A memory instruction that misses every cache takes hundreds of cycles to complete. A warp that issues such a load and then immediately uses the result stops for that entire duration. The processor keeps running only because other warps are resident and can issue in the meantime. Occupancy is the count of those other warps, and it is a budget spent on covering latency, not a score to maximize.
The arithmetic of residency
Residency is capped by a small set of hardware limits. A streaming multiprocessor holds at most a fixed number of blocks, a fixed number of warps, a fixed number of registers across its threads, and a fixed shared-memory capacity that is divided among the resident blocks. Which of these binds first depends on the kernel’s declared resource usage.
Consider a multiprocessor with 65,536 registers available for threads, a maximum of 64 warps, a maximum of 32 blocks, and a shared-memory capacity shared out across the resident blocks. A kernel compiled at 32 registers per thread with 256 threads per block consumes $32 \times 256 = 8192$ registers per block, so eight blocks fit by the register limit. Eight blocks of 256 threads is 2048 threads, or 64 warps, which is exactly the warp limit. Both ceilings are hit at once, and the kernel is at full occupancy.
Raise the register count to 40 per thread. Each block now needs 10,240 registers, and $65536 / 10240$ is 6 blocks. That is 48 warps, or 75% occupancy. The registers did not make the kernel slower by themselves; they removed twelve warps’ worth of latency cover.
The general calculation rounds in a way that catches people out: register allocation is done in granularity units, so a block whose threads use 33 registers each is charged as though they used the next multiple of the allocation granularity. A kernel that appears to use 33 registers may therefore be costed at 40, and the block count it yields can drop by one. Occupancy calculators supplied with the toolchain apply the granularity; hand arithmetic often does not.
Shared memory works the same way with a different granularity: allocations are rounded up to a fixed unit, and the per-block reservation is charged even if the block uses less. A kernel requesting a few hundred bytes per block for a lookup table pays for the rounded allocation, and if the block count is high the cumulative reservation becomes the binding limit rather than registers.
// Asking the runtime for the compiled register count is more reliable
// than trusting a compiler report from a different build.
cudaFuncAttributes attr;
cudaFuncGetAttributes(&attr, my_kernel);
printf("regs/thread=%d smem/block=%zu maxthreads=%d\n",
attr.numRegs, attr.sharedSizeBytes, attr.maxThreadsPerBlock);Latency, throughput, and Little’s law
Occupancy matters only insofar as it produces concurrent memory requests. The relationship is Little’s law, applied to the memory system rather than to a queue:
$$ \text{concurrency} = \text{request rate} \times \text{latency} $$
If the memory system’s latency is $L$ cycles and a kernel needs to sustain $C$ concurrent requests to saturate bandwidth $B$ at a granularity of $S$ bytes per request, then
$$ C = \frac{B \cdot L}{S} $$
Take a bandwidth of roughly 1 TB/s, a latency of roughly 500 cycles at a clock of roughly 1.5 GHz (about 330 ns), and a 128-byte request granularity. Then $C \approx 10^{12} \times 3.3 \times 10^{-7} / 128 \approx 2600$ outstanding requests are needed. With 64 warps resident, that is about 40 outstanding requests per warp. Since a warp can typically have a limited number of loads in flight before it must consume a result, a kernel with one load per iteration and an immediate use cannot reach that figure at any occupancy. This is the reason high occupancy sometimes fails to produce high bandwidth, and the reason a low-occupancy kernel with many independent loads per thread sometimes does.
The implication for tuning is direct. Occupancy is a multiplier on per-warp memory-level parallelism, not a substitute for it. Doubling warps doubles the available concurrency only if each warp continues to have requests outstanding; if the extra warps are stalled at a barrier or on a dependency chain, they contribute nothing. The relationship between per-thread independence and achieved bandwidth is worked through in memory-level parallelism.
Why more occupancy can be slower
Full occupancy is not a target, and several mechanisms make a lower-occupancy kernel faster.
Register spilling. Forcing a kernel under a register ceiling to raise occupancy causes the compiler to spill live values to local memory, which is DRAM-backed. Each spill is a store and each reload a load, so the byte traffic rises. The trade is a well-known one: occupancy gained at the cost of extra memory traffic in a kernel that is already memory bound is a net loss. The roofline position from the roofline bound tells which side of the trade a kernel is on.
Cache capacity pressure. More resident warps means more distinct working sets in flight. If the aggregate working set exceeds L2 capacity, the extra concurrency converts cache hits into DRAM traffic. A kernel whose reuse depends on a tile staying resident in L2 can be slowed by raising occupancy, because the tiles of different blocks evict each other.
Tail effects at block granularity. A high-occupancy configuration with many small blocks produces more scheduling events and a larger tail, which is the subject of wave quantization. The steady-state gain can be smaller than the tail cost.
Shared memory conflicts. Raising occupancy by shrinking the shared-memory footprint often means changing the layout, and a layout that produces bank conflicts costs more than the occupancy gains.
The practical rule is to find the occupancy at which the kernel’s measured rate stops improving as resident warps increase. Beyond that point the extra warps are not buying latency cover, and the resources they consume could go to registers or cache instead.
The occupancy-versus-work-per-thread curve
Sweeping a kernel over block sizes and per-thread work produces a characteristic shape. At small block sizes and low occupancy the rate is low, limited by concurrency. It rises as more warps become resident. At some point it plateaus, because the memory system is saturated. Past the plateau, increasing occupancy further either does nothing or degrades the rate through the mechanisms above.
The plateau’s height is the practical ceiling discussed in achieved versus peak throughput. The plateau’s position on the occupancy axis is the minimum occupancy needed to reach it. A kernel operating to the left of that position has a concurrency problem; one operating on the plateau has a bandwidth or compute problem; one operating to the right of the plateau has a resource contention problem introduced by the tuning itself.
| Block size | Blocks/SM | Warps/SM | Typical position |
|---|---|---|---|
| 64 | Many, block-limited | Low | Left of plateau |
| 128 | Register-limited | Moderate | Near plateau |
| 256 | Register-limited | High | On plateau |
| 512 | Few | High, coarse tail | On plateau, larger tail |
| 1024 | One or two | Maximum, coarsest tail | Right of plateau |
The table is qualitative because the limits differ per architecture, but the shape holds. The useful experiment is to vary block size while holding per-thread work constant, and to plot the achieved rate against the computed occupancy rather than against the block size. Two configurations with the same occupancy but different block sizes should land at the same point on that plot; if they do not, the difference is attributable to block granularity or tail effects rather than to concurrency.
Reading an occupancy report
Profiling tools report both theoretical occupancy, computed from the compiled resource usage, and achieved occupancy, measured as the average number of resident warps over the kernel’s lifetime. The two diverge for reasons that are themselves diagnostic.
A gap between them at kernel start and end is the ramp: blocks do not all begin at once, and the last blocks finish after earlier ones have retired. For short kernels this ramp is a large fraction of the total. A gap in the middle of the kernel indicates blocks retiring unevenly, which usually means a data-dependent branch whose two sides have very different durations, or a grid whose blocks have unequal work. That is a divergence and imbalance problem, not an occupancy problem, and it is diagnosed through warp stall attribution.
An achieved occupancy that is high while the kernel is still slow moves the diagnosis entirely: the warps are present but are not issuing, and the stall reasons say why. Occupancy is the supply of latency-hiding capacity; the stall counters report the demand. Reading only one of the two produces a conclusion that the other would contradict.
References
- CUDA C++ Best Practices Guide — Occupancy — NVIDIA
- CUDA C++ Programming Guide — Compute Capabilities — NVIDIA
- Dissecting the NVIDIA Volta GPU Architecture via Microbenchmarking — Jia, Maggioni, Staiger, and Scarpazza; arXiv