Memory Wall
Definition
The memory wall is the widening gap between how fast compute (peak FLOP/s) scales and how fast memory bandwidth (HBM) scales across GPU generations. Because compute grows far faster than the bandwidth feeding it, the ridge point marches right each generation: a kernel must do more arithmetic per byte to stay compute-bound, so ops that were compute-bound on old hardware silently become memory-bound on new hardware. “Just add more FLOPs” stopped being a strategy.
Key math
| GPU | Year | Peak FP16/BF16 TC | HBM BW | Ridge (FLOP/byte) |
|---|---|---|---|---|
| P100 | 2016 | ~21 TFLOP/s | 720 GB/s | ~29 |
| V100 | 2017 | 125 TFLOP/s | 900 GB/s | ~139 |
| A100 | 2020 | 312 TFLOP/s | 2.0 TB/s | ~156 |
| H100 | 2022 | 990 TFLOP/s | 3.35 TB/s | ~295 |
| B200 | 2024/25 | ~2250 TFLOP/s | ~8 TB/s | ~280 |
P100→H100: compute grew ~47×, bandwidth only ~4.6×. Ridge point climbs from ~29 to ~295 FLOP/byte.
Energy corollary (ratios are the point, ~1 pJ per on-chip FMA):
| Operation | Energy |
|---|---|
| FP16/BF16 multiply-add (on-chip) | ~1 pJ |
| Read from register / SMEM | ~1–10 pJ |
| Read from L2 | ~20–100 pJ |
| Read a word from HBM | ~100–1000× a FLOP |
| Move a word host↔device (PCIe) | ~1000s× a FLOP |
An off-chip access costs 2–3 orders of magnitude more energy than the FLOP it feeds — so minimizing HBM traffic is simultaneously a speed, power, and cost optimization (“IO-aware” design).
Why it matters
As accelerators get faster, a growing fraction of any LLM workload becomes bandwidth-limited. Lower precision (FP8) doubles compute but not bandwidth, pushing the ridge point further right — more ops go memory-bound. The lever that keeps mattering is arithmetic intensity, raised by reuse (tiling, fusion), not faster math. This is the deep reason FlashAttention, fused elementwise chains, and tiled GEMM exist.
Taught in
- 02-gpu-memory-hierarchy — §7 the memory wall and data-movement energy.
See also
- arithmetic-intensity — the lever against the wall
- roofline-model — the ridge point marching right
- gpu-memory-hierarchy — HBM as the scarce resource
- memory-coalescing — using the bandwidth you have
- mixed-precision-training — cutting bytes moved
- gpu-systems-for-llms