GPU Memory Hierarchy
Definition
The GPU memory hierarchy trades capacity for speed across five tiers — registers → shared memory / L1 → L2 → HBM (global) → host DRAM. The fastest storage is tiny and private; the largest is slow and shared. The entire discipline of kernel optimization is: pull data down the pyramid once, reuse it as many times as possible up top, write it back once. Numbers below are H100 (Hopper, SXM5); the shape is constant across generations, only the constants move.
Key math
| Level | Scope | Capacity | Latency | Bandwidth | Managed by |
|---|---|---|---|---|---|
| Registers | per-thread | 256 KB/SM (64K × 4B) | ~1 cycle | ~tens of TB/s per SM | compiler |
| Shared mem / L1 | per-SM | up to 228 KB SMEM (256 KB unified) | ~20–30 cyc | ~19 TB/s aggregate | programmer (SMEM) |
| L2 cache | GPU-wide | 50 MB | ~150–250 cyc | several TB/s | hardware |
| Global / HBM3 | GPU-wide | 80 GB | ~450–800 cyc | 3.35 TB/s | programmer (allocs) |
| Host DRAM | system | 100s GB | ~µs | PCIe5 ~64 GB/s, NVLink4 ~900 GB/s | programmer (copies) |
Two enormous cliffs: register→SMEM is ~20–30×; SMEM→HBM is another ~20–40× latency and ~150× bandwidth.
Latency is hidden, bandwidth is not. A ~500-cycle HBM load is hidden by swapping in other ready warps (occupancy). But if a kernel must move bytes through 3.35 TB/s, no parallelism moves them faster. Latency is a scheduling problem; bandwidth is physics.
Register spills → “local memory”: despite the name, local memory physically lives in HBM (cached in L1/L2). A spill turns a ~1-cycle access into a potential ~500-cycle round trip. Check ptxas -v for spill loads/stores. Registers per thread also gate occupancy.
Why it matters
On modern accelerators a FLOP is nearly free and a byte from HBM is expensive — in time and energy (an off-chip access costs ~100–1000× the FLOP it feeds). Most LLM ops (attention, decode GEMV, all elementwise glue) are bandwidth-bound, so “not touching HBM” is performance engineering. FlashAttention, fused elementwise, tiled GEMM, and KV-cache layout are all schemes to do more work per byte pulled from HBM.
Taught in
- 02-gpu-memory-hierarchy — §3 the hierarchy top to bottom, §4 registers/spills.
- 01-gpu-architecture-and-simt — §4 the on-SM memory.
See also
- memory-coalescing — moving HBM bytes efficiently
- shared-memory-bank-conflicts — the SMEM tier hazard
- memory-wall — why HBM is the scarce resource
- arithmetic-intensity — FLOPs per HBM byte
- occupancy — hiding this latency
- gpu-systems-for-llms