GEMM (General Matrix Multiply)

Definition

GEMM — dense general matrix multiply, $C=\alpha AB+\beta C$ — is the canonical GPU workload and the compute core of transformers. QKV projections, the attention output projection, and the MLP up/down projections are all GEMMs, and across a model they are the large majority of the FLOPs. GEMM is the ideal fit for GPU hardware: embarrassingly parallel, uniform control flow, high arithmetic intensity when tiled, and a direct map onto Tensor Cores.

Key math

For $A\in {\mathbb{R}}^{M\times K}$, $B\in {\mathbb{R}}^{K\times N}$:
$\text{FLOPs}=2MNK,{\text{bytes}}_{\text{ideal}}=(MK+KN+MN)\cdot b$
(each output is a length-$K$ dot product = $2K$ FLOPs; $b$ = bytes/element). For a square GEMM ($M=N=K=n$):
$I_{\text{GEMM}}=\frac{2n^3}{3n^{2}b}=\frac{2n}{3b}$
Intensity grows linearly with $n$ → big GEMMs are compute-bound, small ones (decode-time GEMV, $M=1$) are memory-bound at $I\approx 1$. On H100 BF16 ($I^{\ast }\approx 296$), an $n=4096$ GEMM has $I\approx 1365\gg I^{\ast }$ → firmly compute-bound.

Why GEMM is the perfect GPU workload:

1. Embarrassingly parallel — every output tile independent → fills thousands of ALUs, endless ready warps for latency hiding.
2. Uniform control flow — no data-dependent branching → zero warp divergence.
3. Maps onto Tensor Cores — GEMM is tiled MMA.
4. High intensity when tiled — $O(n^3)$ compute over $O(n^2)$ data → becomes compute-bound.

High-performance implementation (tiling): load tiles of $A,B$ from HBM into shared memory with coalesced reads, reuse each element across an output tile (raising $I$ from $O(1)$ to $O(\text{tile})$), arrange SMEM conflict-free. This recipe takes a matmul from a few % of peak to 80–90%+.

Why it matters

A transformer forward/backward pass is mechanically a long chain of GEMMs with memory-bound elementwise glue between them. Every LLM FLOP estimate (MFU, the 6ND rule) and every performance ceiling is grounded in GEMM. Making GEMM run near peak — in the lowest acceptable precision — is the single highest-leverage lever in training and prefill.

Taught in

• 01-gpu-architecture-and-simt — §9 the GEMM/LLM connection.
• 03-performance-modeling-roofline — §3 counting GEMM FLOPs/bytes, §9 GEMM vs attention.

See also

• tensor-core — the engine GEMM runs on
• arithmetic-intensity — $2n/3b$, the compute-bound lever
• memory-coalescing — coalesced tile loads
• shared-memory-bank-conflicts — conflict-free tiles
• roofline-model — classifying a GEMM
• gpu-systems-for-llms