Kernel Fusion

Definition

Kernel fusion folds a chain of separate GPU kernels into one, so intermediate tensors are produced and consumed on-chip (registers/SMEM) instead of being round-tripped through HBM. It is the primary lever for the memory-bound “glue” between GEMMs — elementwise activations, biases, norms, dropout, residuals — whose runtime is set by HBM traffic, not FLOPs. Fusion changes no math; it removes bytes.

How it works

Every elementwise op has arithmetic intensity $I\approx O(1)$ FLOP/byte — far left of the H100 BF16 ridge (~296) — so each is memory-bound: read the tensor, do trivial math, write it back. A chain of $k$ unfused ops round-trips the tensor $k$ times:
$\text{unfused}={\sum }_{\text{ops}}(\text{read}+\text{write})\gg \text{fused}={\text{read}}_{\text{first}}+{\text{write}}_{\text{last}}.$
For x → bias → GELU → dropout → residual on $n$ BF16 elements, four kernels move $\approx 4\cdot (2n+2n)=16n$ bytes; fused, $\approx 4n$ — a ~4× traffic cut → ~4× speedup on a bandwidth-bound op, for free. Same FLOP numerator, smaller byte denominator → higher $I$, further right on the roofline. A secondary win: one launch instead of $k$ (matters most for tiny tensors where launch cost rivals compute).

Vertical (producer→consumer) fusion fuses along a dependency chain (the bias→GELU→dropout example), eliminating intermediates entirely. This is the high-value case; FlashAttention is an extreme vertical fusion of the whole matmul → softmax → matmul attention chain.

Horizontal fusion fuses independent ops sharing an input or launch — e.g. Q, K, V projections as one batched GEMM, or the same elementwise op over several tensors. The win is amortized launch overhead and better occupancy, not eliminated intermediates.

Fusion does not help compute-bound GEMMs: their runtime is Tensor-Core throughput, so removing their small relative I/O barely moves the needle — hence you fuse around GEMMs (or fuse an elementwise epilogue into the GEMM), not the GEMMs themselves.

Why it matters

The activation glue between GEMMs is where a large chunk of avoidable HBM traffic hides; fusing it is often the biggest single-GPU win after using cuBLAS for the matmuls. Modern stacks automate most of it: torch.compile/TorchInductor auto-fuses pointwise+reduction chains into Triton kernels with zero code changes — though it generally won’t discover a fundamentally new algorithm like FlashAttention, which requires hand IO-aware restructuring.

Taught in

• 06-memory-wall-and-flashattention — §3 vertical/horizontal fusion, the traffic math, torch.compile as autofuser.
• 07-triton-and-modern-kernels — §5 fused softmax; §9 Inductor fusion.

See also

• flash-attention — fusion of the entire attention chain
• memory-wall — why fusion wins
• arithmetic-intensity — fusion raises $I$
• io-aware-algorithms — the design principle
• torch-compile — automatic fusion
• triton — the language fused kernels are written in
• gemm — the compute-bound anchors fusion works around
• gpu-systems-for-llms