Learn: Performance Modeling & Roofline

What you're learning

The quantitative toolkit for reasoning about why a GPU kernel is slow and what the ceiling is — before you ever touch a profiler. By the end you should be able to: count a kernel’s FLOPs and bytes, compute its arithmetic intensity, place it on a roofline, classify it as memory- or compute-bound, pick the right precision, and translate a training run into GPU-hours via MFU and the $6ND$ rule. This is the lens used for the rest of the curriculum.

This lesson assumes 01-gpu-architecture-and-simt (SIMT, occupancy) and 02-gpu-memory-hierarchy (memory hierarchy, HBM bandwidth, the memory wall). We now make the memory wall quantitative.

---

1. Learning map

graph TD
    A["Memory wall<br/>(lesson 02)"] --> B["Count FLOPs<br/>& bytes"]
    B --> C["Arithmetic intensity<br/>I = FLOPs / bytes"]
    C --> D["Roofline model<br/>attainable FLOP/s vs I"]
    D --> E["Ridge point<br/>I* = peak / bandwidth"]
    E --> F["Classify kernel:<br/>memory- vs compute-bound"]
    G["Precision formats<br/>FP32/TF32/BF16/FP8/INT8"] --> H["Tensor cores<br/>throughput by precision"]
    H --> D
    G --> I["Mixed-precision training<br/>master weights + loss scaling"]
    F --> J["MFU / HFU<br/>utilization metrics"]
    I --> J
    K["6ND rule<br/>C ≈ 6ND"] --> J
    J --> L["Training cost<br/>FLOPs → GPU-hours"]
    F --> M["Profiling<br/>Nsight Compute / Systems"]
    L --> N["GPU/LLM connection:<br/>why FP8, what to fuse"]
    F --> N

    style D fill:#4a4,color:#fff
    style C fill:#44a,color:#fff
    style N fill:#a44,color:#fff

---

2. Why performance modeling matters

You cannot optimize what you cannot bound. The single most common failure mode in GPU work is spending a week hand-tuning a kernel that was already at 95% of its physical ceiling — or, worse, chasing FLOP/s on a kernel that is fundamentally memory-bound, where more math throughput does literally nothing.

The roofline model is the antidote: a back-of-envelope upper bound on performance derived from two hardware numbers (peak FLOP/s and peak bandwidth) and one kernel number (arithmetic intensity). It tells you, before profiling, whether a kernel is limited by compute or by memory, and therefore which knob to turn. Horace He’s framing is the canonical mental model: a GPU workload is bottlenecked by compute, memory bandwidth, or overhead — and the first job is always to figure out which.¹

The one question that organizes everything

“Is this kernel memory-bound or compute-bound?” Almost every optimization decision — fusion, precision choice, tiling, recompute vs. store — flows from the answer. The roofline is just the formal way to answer it.

---

3. Counting FLOPs and bytes

FLOPs

A FLOP is one floating-point operation. The convention that matters: a fused multiply-add (FMA), $d\leftarrow a\cdot b+c$, counts as 2 FLOPs (one multiply, one add). This is why matrix products come out to $2MNK$ rather than $MNK$.

For a matrix multiply $C=AB$ with $A\in {\mathbb{R}}^{M\times K}$, $B\in {\mathbb{R}}^{K\times N}$:

$\text{FLOPs}=2MNK$

Each of the $MN$ output elements is a dot product of length $K$ → $K$ multiplies + $K$ adds = $2K$ FLOPs.

Bytes

Bytes = the traffic that must cross the bottleneck memory (almost always HBM/DRAM, per the memory wall). For a GEMM in precision with $b$ bytes/element, in the ideal case where each operand is read once and the output written once:

$\text{bytes}=(\underset{A}{\underbrace{MK}}+\underset{B}{\underbrace{KN}}+\underset{C}{\underbrace{MN}})\cdot b$

"Bytes" means traffic, not footprint

The relevant quantity is bytes moved across the bottleneck, not the size of the tensors. If a value is re-read from HBM 10 times because it didn’t fit in cache/SMEM, that’s 10× the traffic. Conversely, if an intermediate never leaves on-chip memory (the whole point of fusion), it costs zero HBM bytes. A good kernel’s actual traffic can be far above the ideal lower bound — quantifying that gap is what tiling and fusion are about.

Arithmetic intensity

$\boxed{I=\frac{\text{FLOPs}}{\text{bytes}}}\left[\frac{\text{FLOP}}{\text{byte}}\right]$

Arithmetic intensity is the FLOPs performed per byte of HBM traffic. It is a property of the algorithm + its data movement, independent of the hardware. It’s the x-axis of the roofline. Intuitively: high $I$ = you do a lot of math for every byte you fetch (good, you can keep the ALUs fed); low $I$ = you’re mostly shuffling data (bad, the ALUs starve).

For a square GEMM ($M=N=K=n$) in precision $b$:

$I_{\text{GEMM}}=\frac{2n^3}{3n^2\cdot b}=\frac{2n}{3b}$

Note it grows linearly with $n$ — big GEMMs are compute-bound, small ones aren’t. That single fact drives a huge amount of LLM systems design.

Quiz: An elementwise GELU reads a tensor of $n$ FP16 elements and writes $n$ back, doing ~8 FLOPs per element. What is its arithmetic intensity, and does it depend on $n$?

Answer

Bytes $=2n\cdot b=2n\cdot 2=4n$ (read + write, 2 bytes each). FLOPs $\approx 8n$. So
$I=\frac{8n}{4n}=2\text{FLOP/byte},$
independent of $n$. This is the signature of every elementwise op: constant, tiny arithmetic intensity (typically $O(1)$), because you touch each byte a fixed number of times regardless of tensor size. Elementwise ops are therefore always memory-bound on modern GPUs — which is exactly why we fuse them into their neighbors (lesson 06).

---

4. The roofline model

The roofline (Williams, Waterman & Patterson, 2009)² plots attainable FLOP/s (y) against arithmetic intensity (x), both on log-log axes. The attainable performance of a kernel is capped by the minimum of two limits:

$P_{\text{attainable}}(I)=\min (\underset{\text{compute roof}}{\underbrace{P_{\text{peak}}}},\underset{\text{memory roof}}{\underbrace{\beta \cdot I}})$

where $P_{\text{peak}}$ is peak FLOP/s and $\beta$ is peak memory bandwidth (bytes/s). Two regimes:

• Memory-bound region (left): performance $=\beta \cdot I$. It rises with slope $\beta$ (on log-log, slope 1). You’re limited by how fast bytes arrive. More FLOP/s capability is useless; you must either raise $I$ (fusion, better reuse) or move fewer bytes (lower precision, caching).
• Compute-bound region (right): performance $=P_{\text{peak}}$, a flat ceiling. You’re saturating the ALUs/tensor cores. Optimize by using faster math units (tensor cores, lower precision) or improving instruction-level efficiency.

The two lines cross at the ridge point:

$\boxed{I^{\ast }=\frac{P_{\text{peak}}}{\beta }}\left[\frac{\text{FLOP}}{\text{byte}}\right]$

A kernel with $I<I^{\ast }$ is memory-bound; with $I>I^{\ast }$ it is compute-bound. The ridge point is the arithmetic intensity you must exceed to have any hope of saturating the machine.

   attainable FLOP/s  (log)
   ^
P_peak |. . . . . . . . . ._______________________  <- compute roof (flat = peak FLOP/s)
       |                 /:
       |     slope = β  / :
       | (mem. roof)   /  :
       |              /   :
       |             /    :
       |            /     :
       |           /      :
       |__________/_______:_____________________> arithmetic intensity I (log)
                          ^
                       ridge point  I* = P_peak / β
        <-- memory-bound -->|<-- compute-bound -->

Peak vs. achievable FLOP/s — read the fine print

The compute roof is the peak the vendor advertises, and vendors quote the most flattering number: with structured (2:4) sparsity and the lowest precision. NVIDIA’s headline “3,958 TFLOP/s FP8” for H100 is with sparsity; the dense number is ~1,979.³ Real dense kernels never hit peak — a great GEMM reaches ~80–90% of dense peak, and most kernels far less. Always build your roofline from dense peaks in the precision you actually use, and treat the roof as an unreachable asymptote, not a target.

Worked example: H100 SXM roofline

Real numbers for one H100 SXM5 (dense tensor-core peaks, HBM3):³

Quantity	Value
HBM3 bandwidth $\beta$	$3.35\text{ TB/s}=3.35\times 10^{12}$ B/s
Peak BF16 tensor (dense)	$\approx 990$ TFLOP/s $=9.9\times 10^{14}$
Peak FP8 tensor (dense)	$\approx 1979$ TFLOP/s $=1.98\times 10^{15}$
Peak TF32 tensor (dense)	$\approx 495$ TFLOP/s
Peak FP32 (CUDA cores)	$\approx 67$ TFLOP/s

Ridge points:

$I_{\text{BF16}}^{\ast }=\frac{9.9\times 10^{14}}{3.35\times 10^{12}}\approx 296\frac{\text{FLOP}}{\text{byte}},I_{\text{FP8}}^{\ast }=\frac{1.98\times 10^{15}}{3.35\times 10^{12}}\approx 591\frac{\text{FLOP}}{\text{byte}}$

Lower precision raises the ridge point

Halving precision doubles peak FLOP/s but leaves bandwidth (roughly) unchanged, so the ridge point moves right. FP8’s ridge is ~591 FLOP/byte vs BF16’s ~296. This is subtle and important: FP8 makes compute cheaper, which means a larger fraction of kernels become memory-bound. You need even higher arithmetic intensity to stay compute-bound on the faster math units — one more reason fusion and large tile sizes matter more as precision drops.

Quiz: You run a BF16 GEMM with $M=N=K=4096$ on an H100. Compute its arithmetic intensity and classify it. Then do the same for a "GEMV"-like decode step: $M=1$, $K=N=4096$.

Answer

Big GEMM ($n=4096$, BF16 so $b=2$): $I=\frac{2n}{3b}=\frac{2\cdot 4096}{3\cdot 2}=\frac{8192}{6}\approx 1365$ FLOP/byte. Since $1365\gg I_{\text{BF16}}^{\ast }\approx 296$, it is firmly compute-bound — tensor cores are the bottleneck, use them at the lowest acceptable precision.

Decode GEMV ($M=1$): FLOPs $=2MNK=2\cdot 1\cdot 4096\cdot 4096$. Bytes $\approx (MK+KN+MN)b=(4096+4096^2+4096)\cdot 2\approx 2\cdot 4096^2$ (the weight matrix $B$ dominates). So $I\approx \frac{2\cdot 4096^2}{2\cdot 4096^2}=1$ FLOP/byte $\ll 296$: heavily memory-bound. This is the reason LLM autoregressive decoding is bandwidth-limited — batch size 1 means each huge weight matrix is read from HBM to do a single vector’s worth of math. The fix is batching (raises $M$, hence $I$) or keeping weights in a faster tier.

---

5. Precision formats and tensor cores

A floating-point number is sign · mantissa · 2^exponent. Exponent bits set dynamic range (how big/small before overflow/underflow); mantissa bits set precision (relative resolution). The whole precision story is trading these two against bit-width.

Format	Bits	Sign	Exp	Mantissa	Max exp range	Notes
FP32	32	1	8	23	$\sim 10^{\pm 38}$	IEEE single; the reference
TF32	19*	1	8	10	$\sim 10^{\pm 38}$	FP32 range, FP16 precision; stored in 32b reg, tensor-core input
FP16	16	1	5	10	$\sim 10^{\pm 5}$	Precise but narrow range → needs loss scaling
BF16	16	1	8	7	$\sim 10^{\pm 38}$	FP32 range, less precision → won for training
FP8 E4M3	8	1	4	3	$\sim \pm 448$	More precision, less range; used for fwd (weights/acts)4
FP8 E5M2	8	1	5	2	$\sim \pm 57344$	More range, less precision; used for grads4
INT8	8	—	—	—	$[-128,127]$	Integer + scale factor; inference quantization

*TF32 uses 19 significant bits internally but occupies a 32-bit register slot.

Why BF16 beat FP16 for training

FP16 has only 5 exponent bits → dynamic range $\sim 6\times 10^{-5}$ to $65504$. Gradients routinely underflow to zero in that range, which is why FP16 training requires loss scaling. BF16 keeps all 8 of FP32’s exponent bits — it is literally FP32 with the bottom 16 mantissa bits chopped off. Same range as FP32, so gradients don’t underflow and you usually need no loss scaling. You pay in precision (7 mantissa bits), but neural-net training is robust to that. Truncating/rounding FP32↔BF16 is also trivial (drop 16 bits). That combination — no range surprises, cheap conversion — is why BF16 is the default training dtype.⁵

Tensor cores

Tensor cores are dedicated units that compute a small matrix-multiply-accumulate (MMA), $D=AB+C$, on a tile (e.g. $16\times 8\times 16$) in a handful of cycles — this is where the $10\times$+ over FP32 CUDA cores comes from. Crucially, tensor cores typically accumulate in FP32 (or FP16) even when inputs are BF16/FP8, which is what makes low-precision training numerically viable: the products are low-precision but the running sum keeps FP32 precision.⁶

H100 dense tensor-core throughput roughly doubles each time you halve input precision:³

Precision	Dense TFLOP/s	vs BF16
TF32	~495	0.5×
BF16 / FP16	~990	1×
FP8 (E4M3/E5M2)	~1979	2×
INT8	~1979 (TOPS)	2×

(All roughly double again with 2:4 structured sparsity — the marketing numbers.)

Mixed-precision training

The standard recipe (AMP-style), which lets you get tensor-core speed without wrecking convergence:⁷

1. Master weights in FP32. The optimizer keeps a high-precision copy. Updates are $w\leftarrow w-\eta g$; if $\eta g$ is tiny relative to $w$, adding it in BF16 would round to a no-op (the “swamping” problem). FP32 master weights preserve those small updates.
2. Compute in low precision. Cast weights/activations to BF16 (or FP8) for the forward and backward GEMMs — this is where the speedup lives.
3. Loss scaling (needed for FP16, usually not BF16). Multiply the loss by $S$ before backward so small gradients land in representable range; divide gradients by $S$ before the optimizer step. Dynamic loss scaling adjusts $S$ up/down to avoid overflow.
4. FP32 accumulation inside the tensor-core MMA and for reductions (softmax, layernorm, loss).

Quiz: You train in BF16 with FP32 master weights. Why can't you just keep the weights in BF16 and skip the FP32 copy to save memory?

Answer

Because of update swamping. With 7 mantissa bits, BF16 has relative resolution $\sim 2^{-8}\approx 0.4\%$. Late in training a weight might be $\sim 1.0$ while its update $\eta g$ is $\sim 10^{-4}$. In BF16, $1.0+10^{-4}$ rounds back to $1.0$ — the update is silently dropped and learning stalls. The FP32 master copy has 23 mantissa bits, so it accumulates these small updates faithfully; you only round to BF16 for the forward/backward math, never for the accumulation of updates. (This is also why optimizer states like Adam’s moments are kept in FP32.)

---

6. MFU and HFU

Peak FLOP/s is what the silicon can do; utilization is what your run actually achieves. Two metrics:

MFU (Model FLOPs Utilization) — fraction of peak spent on the useful model FLOPs (the $6ND$ math that defines the model), ignoring any redundant work:

$\text{MFU}=\frac{\text{model FLOPs per second}}{\text{peak FLOP/s (aggregate)}}=\frac{6N\cdot (\text{tokens/s})}{G\cdot P_{\text{peak}}}$

where $N$ = params, $G$ = number of GPUs, $P_{\text{peak}}$ = dense peak per GPU in the training precision.

HFU (Hardware FLOPs Utilization) — fraction of peak spent on all FLOPs actually executed, including activation-recomputation (gradient checkpointing), which redoes forward FLOPs in the backward pass. HFU counts that redone work; MFU does not.

$\text{HFU}=\frac{\text{hardware FLOPs per second (incl. recompute)}}{G\cdot P_{\text{peak}}}\ge \text{MFU}$

Do not confuse MFU and HFU

HFU $\ge$ MFU, always. With activation recomputation you might see HFU 60% but MFU only 45% — the extra 15 points are wasted forward passes that don’t advance the model, they just trade compute for activation memory. MFU is the number you care about for training efficiency and cost (it maps directly to tokens-per-dollar); HFU tells you how hard the ALUs are working including overhead you introduced. Reporting HFU as if it were MFU flatters your run. The PaLM paper introduced MFU precisely to have a hardware- and recompute-agnostic efficiency number.⁸

Typical real-world MFU: ~35–55% for large-scale transformer training on well-tuned stacks (PaLM reported 46% at 540B⁸; strong Megatron/H100 runs land in the 40s–low 50s). What drags MFU down:

• Memory-bound ops (attention softmax, layernorm, elementwise, embeddings) that don’t use tensor cores.
• Communication — all-reduce / all-gather for data/tensor/pipeline parallelism not overlapped with compute.
• Pipeline bubbles, load imbalance, small batch or short sequences (GEMMs too small → left of the ridge).
• Kernel launch overhead and Python/host-side stalls (fixable with CUDA graphs / fusion).
• Recompute (this is exactly the MFU/HFU gap).

Quiz: A 70B model trains at 3,000 tokens/s/GPU on H100s in BF16. What's the MFU?

Answer

Model FLOPs/s per GPU $=6N\times$ tokens/s $=6\times 70\times 10^9\times 3000=1.26\times 10^{15}$ FLOP/s. Peak dense BF16 $=9.9\times 10^{14}$ FLOP/s.
$\text{MFU}=\frac{1.26\times 10^{15}}{9.9\times 10^{14}}\approx 1.27\Rightarrow 127\%?!$
That’s impossible — a red flag that the inputs are inconsistent (3,000 tok/s/GPU for a 70B model is far too high; realistic is a few hundred). The lesson: MFU is a great sanity check. If you compute MFU > ~60% you almost certainly have a bug in your FLOP count, token count, or peak number. Recompute with, say, 900 tok/s/GPU → $6\cdot 70\text{e}9\cdot 900=3.78\text{e}14$, MFU $=3.78\text{e}14/9.9\text{e}14\approx 38\%$, which is believable.

---

7. The $6ND$ rule

The most useful heuristic in the whole toolkit (Kaplan et al. 2020⁹; Chinchilla / Hoffmann et al. 2022¹⁰): the total training compute of a dense transformer is

$\boxed{C\approx 6ND}$

where $N$ = non-embedding parameters, $D$ = training tokens, $C$ = FLOPs.

Where the 6 comes from. For each parameter, each token requires:

• Forward: the parameter participates in one multiply-add per token → $2$ FLOPs. ($2N$ FLOPs/token.)
• Backward: roughly twice the forward cost — one pass to compute the gradient w.r.t. the input (to propagate) and one w.r.t. the weight → $4$ FLOPs. ($4N$ FLOPs/token.)

Total $6$ FLOPs per parameter per token → $C\approx 6ND$. This ignores attention’s $O({\text{seq}}^2)$ term, which is a small correction unless sequences are very long relative to $d_{\text{model}}$.

$6ND$ is the backbone of every training estimate

Cost, time, carbon, cluster sizing, and MFU all derive from $C=6ND$. Memorize it. Its inverse — divide measured throughput by $6N$ to get tokens/s, or by peak to get MFU — is how you sanity-check any training log in ten seconds.

---

8. Profiling: Nsight Systems vs Nsight Compute

The roofline tells you the ceiling; profilers tell you where you are. NVIDIA gives two complementary tools — know which question each answers.

graph LR
    subgraph "Nsight Systems (nsys) — timeline / macro"
        A["Full timeline across GPUs+CPU"]
        B["Kernel/stream/NVTX ranges"]
        C["Gaps, launch overhead,<br/>CPU stalls, comm overlap"]
    end
    subgraph "Nsight Compute (ncu) — single kernel / micro"
        D["One kernel, deep dive"]
        E["SM & memory throughput %,<br/>occupancy, warp stalls"]
        F["Built-in roofline chart"]
    end
    A --> G["Which kernel/gap<br/>dominates? Is comm<br/>overlapped?"]
    D --> H["Is THIS kernel mem-<br/>or compute-bound?<br/>How close to roof?"]

• Nsight Systems (nsys) — the timeline / whole-program view. Shows every kernel, memcpy, and CUDA/NCCL call on a time axis across all streams and GPUs. Use it first to answer: where is time going? Find the dominant kernel, spot gaps (launch overhead, host stalls, un-overlapped all-reduce), check whether compute and communication overlap. It’s how you find the overhead-bound problems that the roofline can’t see.
• Nsight Compute (ncu) — the single-kernel microscope. Replays one kernel with hardware counters and reports SM throughput %, memory (DRAM) throughput %, achieved occupancy, warp-stall reasons, and — critically — a built-in roofline chart that plots the kernel’s measured $(I,\text{FLOP/s})$ dot against the compute and memory roofs. If the dot sits on the memory roof, you’re bandwidth-bound; on the flat roof, compute-bound; below both, you have an occupancy/latency/overhead problem.

The two-step profiling workflow

Always nsys first to find which kernel matters (don’t optimize a kernel that’s 2% of runtime), then ncu on that kernel to find why it’s slow and how far it is from its roofline roof. SM-throughput-% high + memory-% low → compute-bound; the reverse → memory-bound; both low → latency/occupancy-bound (too few warps to hide latency — back to lesson 01).

---

9. GPU/LLM connection

Estimating training cost and time

Combine $C=6ND$ with MFU to turn a model spec into wall-clock and dollars. Effective throughput per GPU is $\text{MFU}\cdot P_{\text{peak}}$, so:

$\text{GPU-hours}=\frac{C}{\text{MFU}\cdot P_{\text{peak}}\cdot 3600}=\frac{6ND}{\text{MFU}\cdot P_{\text{peak}}\cdot 3600}$

$\text{wall-clock}=\frac{\text{GPU-hours}}{G}$

Worked estimate — a 70B model on 15T tokens, H100 BF16, MFU 40%, 8192 GPUs:

$C=6\cdot 70\text{e}9\cdot 15\text{e}12=6.3\times 10^{24}\text{FLOPs}$
$\text{eff. per GPU}=0.40\cdot 9.9\times 10^{14}=3.96\times 10^{14}\text{FLOP/s}$
$\text{GPU-hours}=\frac{6.3\times 10^{24}}{3.96\times 10^{14}\cdot 3600}\approx 4.4\times 10^6\text{GPU-hours}$
$\text{wall-clock}=\frac{4.4\times 10^6}{8192}\approx 540\text{hours}\approx 22.5\text{days}$

At ~$2/GPU-hour that’s ~$8.8M of compute — and this matches the ballpark of published Llama-3-70B figures (~6.4M GPU-hours).¹¹ This is the single most important calculation for planning a training run.

Why FP8 matters for frontier scale

The estimate above scales inversely with $P_{\text{peak}}$. Moving from BF16 (~990 TFLOP/s) to FP8 (~1979 TFLOP/s) halves the compute roof’s cost for the tensor-core-bound GEMMs — potentially ~2× fewer GPU-hours for the matmul-dominated part of training. FP8 also halves HBM traffic and NVLink/network bytes for those tensors, which helps the memory- and comm-bound parts too. At frontier scale ($C\sim 10^{25}$–$10^{26}$), that’s the difference between a 3-week and a 6-week run, and tens of millions of dollars. The catch (from §5): FP8’s ridge point is ~591 FLOP/byte, so more ops fall into the memory-bound region, and FP8’s tiny mantissa demands per-tensor scaling to stay numerically stable (the reason NVIDIA’s Transformer Engine exists).¹² This is why FP8 training is a frontier-lab capability, not a free switch.

Which LLM ops are memory- vs compute-bound (and why it motivates fusion)

Placing the transformer’s core ops on the roofline explains the entire kernel-optimization agenda:

Op	Arithmetic intensity	Regime	Why
Large GEMMs (QKV proj, MLP, output)	High ($I\sim 2n/3b\gg I^{\ast }$)	Compute-bound	Big matrices reuse each byte many times → tensor cores saturate
Attention (naive)	Low	Memory-bound	Materializes the $s\times s$ scores to HBM; softmax is elementwise
Attention (FlashAttention)	Higher, near/above ridge	Compute-bound-ish	Keeps scores in SRAM, never writes $s\times s$ to HBM → traffic collapses, $I$ rises
Softmax / LayerNorm / RMSNorm	$O(1)$	Memory-bound	Read + write, almost no math per byte
Elementwise (GELU, residual add, dropout, scaling)	$\sim 1$–$2$	Memory-bound	Fixed FLOPs per byte, independent of size

The pattern: GEMMs are compute-bound; everything gluing them together is memory-bound. A naive stack launches a separate kernel for each memory-bound op, and each one pays a full HBM round-trip (read inputs, write outputs) for almost no math. The fix is fusion (lesson 06): fold the memory-bound ops into their neighboring GEMM or into each other so intermediates stay in registers/SRAM and never touch HBM. FlashAttention is the flagship example — it fuses the entire attention computation to avoid the $s\times s$ HBM write, converting a memory-bound op into a compute-bound one.¹³ The roofline is why fusion is the highest-leverage kernel optimization in LLM training, and it motivates the whole kernel tier of this curriculum.

---

Practice problems

Practice problem 1: Arithmetic intensity of GEMM vs attention — classify each

Problem: On an H100 (BF16, $b=2$, $I^{\ast }\approx 296$), consider a transformer layer with $d_{\text{model}}=8192$, sequence length $s=8192$, and a single head-group. (a) For the MLP up-projection GEMM ($M=s$, $K=d$, $N=4d$), compute $I$ and classify. (b) For attention without FlashAttention, where the $s\times s$ score matrix is written to and read from HBM, estimate $I$ and classify. (c) State what changes with FlashAttention.

Worked solution:
(a) MLP GEMM. $M=8192,K=8192,N=32768$. FLOPs $=2MNK=2\cdot 8192\cdot 32768\cdot 8192\approx 4.4\times 10^{12}$. Bytes $=(MK+KN+MN)b=(8192\cdot 8192+8192\cdot 32768+8192\cdot 32768)\cdot 2\approx (6.7\text{e}7+2.68\text{e}8+2.68\text{e}8)\cdot 2\approx 1.2\times 10^9$. $I\approx 4.4\text{e}12/1.2\text{e}9\approx 3600$ FLOP/byte $\gg 296$ → compute-bound. (Consistent with $I\approx 2n/3b$ for large tiles.)

(b) Naive attention. Dominant traffic is the score matrix $S=Q{K}^{\top }\in {\mathbb{R}}^{s\times s}$: written after the first GEMM, read back for softmax, read again for $SV$. That’s $\sim 3s^{2}b=3\cdot 8192^2\cdot 2\approx 4.0\times 10^8$ bytes just for $S$. FLOPs $\approx 4s^{2}d=4\cdot 8192^2\cdot 8192\approx 2.2\times 10^{12}$. Naively $I\approx 2.2\text{e}12/4.0\text{e}8\approx 5500$? — but the softmax itself is a separate memory-bound kernel ($I\sim 1$) reading/writing the full $s\times s$ matrix, and at long $s$ the $s\times s$ HBM traffic dwarfs reuse. The effective intensity of the attention block as launched (multiple kernels, each round-tripping $S$) collapses toward the memory-bound regime; the softmax kernel in particular sits on the memory roof. Memory-bound in practice.

(c) FlashAttention tiles $Q,K,V$ and computes softmax online in SRAM, so $S$ is never written to HBM. HBM traffic drops to reading $Q,K,V$ and writing $O$: $\sim 4sdb=4\cdot 8192\cdot 8192\cdot 2\approx 1.1\times 10^8$ bytes, while FLOPs stay $\sim 2.2\times 10^{12}$ → $I\approx 2\times 10^4$ FLOP/byte, firmly compute-bound. Same math, ~4× less HBM traffic, and it crosses the ridge point — the roofline made the win predictable.

Practice problem 2: GPU-hours to train an $N$-param model on $D$ tokens at a given MFU

Problem: Estimate the GPU-hours and 30-day feasibility of training a 400B-parameter model on 8T tokens using H100s in FP8 at an optimistic MFU of 35%. How many GPUs to finish in 30 days? Then redo the GPU-hours in BF16 and comment.

Worked solution:
$C=6ND=6\cdot 4\text{e}11\cdot 8\text{e}12=1.92\times 10^{25}$ FLOPs.
FP8 dense peak $=1.979\times 10^{15}$ FLOP/s; effective $=0.35\cdot 1.979\text{e}15=6.93\times 10^{14}$ FLOP/s.
$\text{GPU-hours}=\frac{1.92\text{e}25}{6.93\text{e}14\cdot 3600}\approx 7.7\times 10^6\text{GPU-hours}.$
To finish in 30 days $=720$ h: $G=7.7\text{e}6/720\approx 10,700$ GPUs. So ~10–11k H100s for a month — a frontier-scale cluster.

BF16 version: peak halves to $9.9\text{e}14$, effective $=0.35\cdot 9.9\text{e}14=3.47\text{e}14$, giving $\approx 1.54\times 10^7$ GPU-hours — 2× more than FP8, i.e. ~21,000 GPUs for the same 30 days (assuming you sustain the same MFU, which is itself harder in FP8). This factor-of-2 on a multi-million-GPU-hour run is exactly why frontier labs invest heavily in stable FP8 training.

---

Practice & resources

Hands-on

1. Profile a kernel and read its roofline (Nsight Compute). Write (or grab) a simple SGEMM and an elementwise kernel. Run ncu --set full ./kernel, open the Roofline section in the GUI, and confirm the GEMM’s dot sits near the compute roof while the elementwise dot sits on the memory roof. Cross-check against your hand-computed $I$. Then bump the GEMM to tensor cores (cuBLAS / WMMA) and watch the compute roof and the dot both move.
2. Compute MFU from a real training log. Take any run’s tokens/s, model N, GPU count G, and precision peak. Compute $\text{MFU}=6N\cdot (\text{tok/s})/(G\cdot P_{\text{peak}})$. Sanity-check it’s in ~35–55%; if it’s >60% or >100%, hunt the bug (usually wrong $N$, embedding params double-counted, or a sparse-peak $P_{\text{peak}}$). Then estimate the run’s total GPU-hours with $6ND$ and compare to what you were billed.
3. Timeline pass (Nsight Systems). Run nsys profile on a few training steps. Find the single dominant kernel and check whether NCCL all-reduce overlaps with compute or shows up as a gap. Estimate how much MFU you’d recover by hiding it.

Real resources

• Roofline (primary): Williams, Waterman & Patterson, “Roofline: An Insightful Visual Performance Model for Multicore Architectures,” CACM 2009 — the original paper. dl.acm.org/doi/10.1145/1498765.1498785
• Hardware numbers: NVIDIA H100 Tensor Core GPU Architecture Whitepaper (V1.01) — the definitive per-precision dense/sparse throughput and HBM3 bandwidth tables. resources.nvidia.com/en-us-hopper-architecture and the Hopper deep-dive blog developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth
• $6ND$ / scaling: Kaplan et al., “Scaling Laws for Neural Language Models” (arXiv:2001.08361) — Appendix derives the FLOP count; Hoffmann et al., “Training Compute-Optimal LLMs” (Chinchilla, arXiv:2203.15556).
• The mental model (must-read): Horace He, “Making Deep Learning Go Brrrr From First Principles” — compute vs memory-bandwidth vs overhead, and why fusion wins. horace.io/brrr_intro.html
• MFU: Chowdhery et al., PaLM (arXiv:2204.02311) §Training Efficiency — defines and reports MFU (46% at 540B) and contrasts with HFU.
• Mixed precision: Micikevicius et al., “Mixed Precision Training” (arXiv:1710.03740) — master weights + loss scaling; and NVIDIA’s FP8 formats paper “FP8 Formats for Deep Learning” (arXiv:2209.05433).
• Lectures: GPU MODE (formerly CUDA MODE) lecture series — see the roofline, Nsight, and FlashAttention lectures. github.com/gpu-mode/lectures
• FlashAttention (preview of fusion): Dao et al. (arXiv:2205.14135) — the canonical memory-bound → compute-bound transformation.

---

What’s next

Two threads, in order:

1. 04-first-cuda-kernels (next) — now that you can predict a kernel’s ceiling and classify it, write kernels and measure how close you get. You’ll build the elementwise and GEMM kernels you just roofline-analyzed, and read their ncu rooflines for real.
2. 06-memory-wall-and-flashattention (ahead) — the payoff of this lesson: fusing memory-bound ops so intermediates never hit HBM. The roofline is the why; fusion is the how.
3. Revisit 02-gpu-memory-hierarchy — reread the memory wall now that “bytes” means HBM traffic and you can quantify exactly what a cache miss or an un-fused op costs.

---

Reference pages: roofline-model, arithmetic-intensity, mfu, mixed-precision-training | Topic: gpu-systems-for-llms | Filed: 2026-09-02

---

References

Footnotes

1. Horace He, “Making Deep Learning Go Brrrr From First Principles” (2022). Frames every workload as compute-bound, memory-bandwidth-bound, or overhead-bound. https://horace.io/brrr_intro.html ↩

2. Samuel Williams, Andrew Waterman & David Patterson, “Roofline: An Insightful Visual Performance Model for Multicore Architectures,” Communications of the ACM 52(4):65–76, April 2009. DOI 10.1145/1498765.1498785. https://dl.acm.org/doi/10.1145/1498765.1498785 ↩

3. NVIDIA H100 Tensor Core GPU Architecture Whitepaper (v1.01) and the H100 datasheet. Final shipping H100 SXM5 dense peaks: FP8 1,979 TFLOP/s (3,958 with 2:4 sparsity), BF16/FP16 989 TFLOP/s, TF32 494.5 TFLOP/s, FP32 67 TFLOP/s; 80 GB HBM3 at 3.35 TB/s. (The whitepaper lists rounded preliminary estimates — 2000/1000/500/60 — later refined in the datasheet.) https://resources.nvidia.com/en-us-tensor-core and https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/ ↩ ↩² ↩³

4. Paulius Micikevicius et al. (NVIDIA, Arm, Intel), “FP8 Formats for Deep Learning,” arXiv:2209.05433 (2022). Specifies E4M3 (max normal ±448, recommended for weights/activations) and E5M2 (max normal ±57344, IEEE-style, recommended for gradients). https://arxiv.org/abs/2209.05433 ↩ ↩²

5. Dhiraj Kalamkar et al., “A Study of BFLOAT16 for Deep Learning Training,” arXiv:1905.12322 (2019). BF16 has the same 8 exponent bits as FP32 (identical dynamic range) with a 7-bit mantissa, avoiding the underflow/loss-scaling issues of FP16. https://arxiv.org/abs/1905.12322 ↩

6. NVIDIA Hopper (H100) Architecture Whitepaper — 4th-gen Tensor Cores compute matrix-multiply-accumulate with FP32/FP16 accumulation for low-precision inputs; and the NVIDIA CUDA C++ Programming Guide / PTX ISA “Warp Matrix Functions (WMMA)” and mma documentation for the per-instruction MMA tile shapes (e.g. m16n8k16). https://docs.nvidia.com/cuda/parallel-thread-execution/ and https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/ ↩

7. Paulius Micikevicius et al., “Mixed Precision Training,” arXiv:1710.03740 (ICLR 2018). Introduces the FP32 master-weights + loss-scaling recipe with FP32 accumulation. https://arxiv.org/abs/1710.03740 ↩

8. Aakanksha Chowdhery et al., “PaLM: Scaling Language Modeling with Pathways,” arXiv:2204.02311 (2022), Appendix B. Introduces Model FLOPs Utilization (MFU) as an implementation-independent efficiency metric distinct from Hardware FLOPs Utilization (HFU); reports PaLM 540B at 46.2% MFU (57.8% HFU). https://arxiv.org/abs/2204.02311 ↩ ↩²

9. Jared Kaplan et al., “Scaling Laws for Neural Language Models,” arXiv:2001.08361 (2020). Appendix derives the $C\approx 6ND$ training-FLOPs estimate. https://arxiv.org/abs/2001.08361 ↩

10. Jordan Hoffmann et al., “Training Compute-Optimal Large Language Models” (Chinchilla), arXiv:2203.15556 (2022). Uses the $C=6ND$ compute model to derive compute-optimal $N$/$D$ scaling. https://arxiv.org/abs/2203.15556 ↩

11. Aaron Grattafiori et al. (Meta), “The Llama 3 Herd of Models,” arXiv:2407.21783 (2024). Reports pre-training GPU-hours on H100-80GB (Llama-3 70B ≈ 6.4M GPU-hours). https://arxiv.org/abs/2407.21783 ↩

12. NVIDIA Transformer Engine documentation — library for FP8 training/inference on Hopper that manages per-tensor scaling factors to keep FP8 numerically stable. https://docs.nvidia.com/deeplearning/transformer-engine/ ↩

13. Tri Dao et al., “FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness,” arXiv:2205.14135 (2022). Tiles attention and computes softmax online in SRAM so the $s\times s$ score matrix never touches HBM. https://arxiv.org/abs/2205.14135 ↩