Arithmetic Intensity

Definition

Arithmetic intensity $I$ is the number of FLOPs performed per byte of HBM traffic — a property of the algorithm and its data movement, independent of hardware. It is the x-axis of the roofline and the central lever deciding whether a kernel is memory-bound (low $I$, ALUs starve) or compute-bound (high $I$, ALUs saturate).

Key math

$\boxed{I=\frac{\text{FLOPs}}{\text{bytes}}}\left[\frac{\text{FLOP}}{\text{byte}}\right]$

“Bytes” means traffic across the bottleneck, not footprint. A value re-read from HBM 10× costs 10× the bytes; an intermediate that never leaves on-chip memory (fusion) costs zero HBM bytes. FMA counts as 2 FLOPs.

Square GEMM ($M=N=K=n$, precision $b$ bytes):
$I_{\text{GEMM}}=\frac{2n^3}{3n^{2}b}=\frac{2n}{3b}$
Grows linearly with $n$ — big GEMMs are compute-bound, small ones aren’t. That single fact drives huge amounts of LLM systems design.

Elementwise op (read + write, ~fixed FLOPs/element): e.g. GELU reads/writes $n$ FP16 elements ($4n$ bytes) doing ~$8n$ FLOPs → $I=8n/4n=2$, independent of $n$. The signature of every elementwise op: constant, tiny $I\Rightarrow$ always memory-bound.

Op	Intensity	Regime
Large GEMM (proj, MLP)	$\sim 2n/3b\gg I^{\ast }$	compute-bound
Decode GEMV ($M=1$)	$\sim 1$	memory-bound
Softmax / LayerNorm	$O(1)$	memory-bound
Elementwise (GELU, add)	$\sim 1$–$2$	memory-bound

Why it matters

Classify against the ridge point $I^{\ast }=P_{\text{peak}}/\beta$ (~296 FLOP/byte on H100 BF16): $I<I^{\ast }\Rightarrow$ memory-bound. The pattern in a transformer — GEMMs are compute-bound; everything gluing them together is memory-bound — is why fusion is the highest-leverage kernel optimization. You raise $I$ by reuse (tiling, fusion, larger batch/tile), not by faster math. As the memory wall pushes $I^{\ast }$ right, raising $I$ matters more each generation.

Taught in

• 03-performance-modeling-roofline — §3 counting FLOPs/bytes, arithmetic intensity.
• 02-gpu-memory-hierarchy — §7 the memory wall.

See also

• roofline-model — where $I$ is the x-axis
• memory-wall — ridge point marching right
• gemm — $I=2n/3b$
• gpu-memory-hierarchy — the bytes being counted
• mfu — the run-level utilization number
• gpu-systems-for-llms