Occupancy

Definition

Occupancy is the ratio of active warps resident on an SM to the maximum warps the SM supports (64 on H100 = 2048 threads). It is the primary knob for latency hiding: more resident warps → a deeper pool of ready work → the scheduler more often has something to issue while other warps stall on memory. Crucially, 100% occupancy is not the goal — it must be high enough to saturate the pipeline and no higher.

Key math

Achievable occupancy is the minimum across three per-SM resource ceilings:

1. Registers. With $R$ registers/thread and 65,536 regs/SM: max resident threads $\le \lfloor 65536/R\rfloor$. $R=32\Rightarrow 2048$ threads = 64 warps = 100%; $R=64\Rightarrow$ 32 warps = 50%; $R=128\Rightarrow$ 25%.
2. Shared memory per block. With $S$ bytes/block and up to 228 KB/SM: at most $\lfloor 228\text{KB}/S\rfloor$ blocks fit.
3. Hard caps: 64 resident warps and 32 resident blocks per SM. Tiny blocks (1 warp) hit the 32-block cap at only 32 warps → 50%.

${\text{warps}}_{\text{active}}=\min \left(\lfloor \frac{65536}{R}\rfloor /32,\lfloor \frac{228\text{KB}}{S}\rfloor \cdot \text{blockWarps},64,32\cdot \text{blockWarps}\right)$

Use cudaOccupancyMaxActiveBlocksPerMultiprocessor / the Occupancy Calculator rather than hand arithmetic (register allocation is granularized).

Little’s Law is the real principle: requests needed in flight to saturate bandwidth $N=\text{latency}\times \text{throughput}$. Occupancy is one way to generate that in-flight parallelism (more warps); ILP within a thread (many independent FMAs) is another — which is why low-occupancy, high-ILP kernels can still saturate the GPU.

Why it matters

The occupancy ≠ speed myth. Pushing to 100% often means cutting registers, forcing the compiler to spill to local memory (which lives in HBM) — trading a scheduling gain for a bandwidth cost. Many of the fastest kernels (cuBLAS GEMM tiles, FlashAttention) run at 30–60% occupancy on purpose: they hoard registers/SMEM to keep a large matmul tile on-chip and hide latency via ILP.

For LLM training, this is why maximizing occupancy is the wrong reflex — profile stall reasons and check whether the kernel is already compute- or memory-bound first.

Taught in

• 01-gpu-architecture-and-simt — §7 occupancy and the 100%≠optimal nuance.
• 02-gpu-memory-hierarchy — §4 register pressure → occupancy.

See also

• warp-simt — the resident unit counted
• streaming-multiprocessor — the resource ceilings
• gpu-memory-hierarchy — register file, spills to local memory
• roofline-model — occupancy vs the actual bound
• gpu-systems-for-llms