Streaming Multiprocessor (SM)

Definition

The Streaming Multiprocessor (SM) is the fundamental unit of scheduling and resource allocation on an NVIDIA GPU — a small, independent, massively-multithreaded processor with its own register file, shared memory / L1, warp schedulers, and execution units (CUDA cores, Tensor Cores, SFUs, load/store). A thread block is assigned to exactly one SM. The whole GPU is “just $N$ of these running independently” (H100: 132), so almost all performance reasoning happens at the granularity of one SM.

How it works

Anatomy of one H100 (Hopper, GH100) SM:

• 4 sub-partitions (processing blocks), each with one warp scheduler + dispatch unit, a register-file slice, and its own execution units.
• 128 FP32 CUDA cores (32/sub-partition), 64 FP64, 64 INT32.
• 4 Tensor Cores (4th-gen, one per sub-partition) — the matmul engines.
• SFUs for transcendentals ($\exp$, $\sin$, $1/\sqrt{x}$ — used by softmax, GELU), load/store units.
• 65,536 32-bit registers (256 KB) — the resource that most often gates occupancy.
• 256 KB unified L1/shared memory, up to 228 KB configurable as programmer-managed SMEM.

Each cycle a warp scheduler picks a ready resident warp and issues its next instruction; if that warp stalls on a ~500-cycle HBM load, it issues from another next cycle. This zero-overhead warp switch is the latency-hiding mechanism — no register save/restore, because every resident warp’s registers are physically present the whole time (the reason the register file is huge). Hard per-SM caps: 64 resident warps (2048 threads), 32 resident blocks.

GH100 physically has 144 SMs; 132 are enabled on the SXM5 part (rest disabled for yield). At ~1.98 GHz boost: peak FP32 $=132\times 128\times 2\times 1.98\text{e}9\approx 67$ TFLOP/s (the $\times 2$ is FMA).

Why it matters

Keeping one SM busy is the hard part; scaling to the full chip is trivial. Every occupancy calculation, register/SMEM budget, and Tensor-Core feeding problem in LLM kernels is framed per-SM. The register file and SMEM capacity per SM directly set how large a GEMM or attention tile you can hold on-chip.

Taught in

• 01-gpu-architecture-and-simt — §4 the hardware hierarchy, §7 occupancy.
• 02-gpu-memory-hierarchy — §5 the per-SM SMEM/L1 block.

See also

• cuda-execution-model — the block→SM mapping
• warp-simt — what the schedulers issue
• occupancy — resident warps per SM
• tensor-core — the SM’s matmul units
• gpu-memory-hierarchy — the SM’s register file and SMEM
• gpu-systems-for-llms