Prefill vs. Decode

Definition

LLM inference splits into two phases with opposite performance profiles. Prefill processes the whole prompt at once (a big GEMM), produces the first token, and populates the KV cache — it is compute-bound. Decode generates the rest one token at a time, autoregressively — it is memory-bandwidth-bound. Nearly every serving optimization is “make decode cheaper.”

Key math / How it works

Prefill. All $s_{\text{prompt}}$ tokens pass through the model in parallel, so each weight matrix multiplies an activation of $s_{\text{prompt}}$ rows. Weight bytes (read once) are amortized over many rows → high arithmetic intensity → compute-bound, saturating tensor cores. Happens once.

Decode. Each step feeds a single new token per sequence — a skinny GEMM/GEMV. You re-read the entire model’s weights from HBM to do little math, so intensity $\sim b$ (batch) and is tiny → you sit on the memory roof (roofline). Autoregressive: cannot be parallelized across time within a sequence. Happens once per output token.

${\text{intensity}}_{\text{prefill}}\sim s_{\text{prompt}},{\text{intensity}}_{\text{decode}}\sim b$

The asymmetry. A 500-token reply is ~1 prefill + ~500 decode steps, so decode dominates cost and latency. Since decode is memory-bound, the levers are batching (raise $b$ → raise intensity → throughput, via continuous-batching) and moving fewer bytes (quantization, KV-cache management, speculative-decoding).

Serving metrics.

• TTFT (time to first token) — dominated by prefill; sets perceived responsiveness.
• TPOT / ITL (time per output token / inter-token latency) — dominated by decode; sets streaming speed.
• Throughput (tok/s across requests) — the cost metric; traded against latency via batch size.

Why it matters

The prefill/decode split is the organizing fact of serving: it explains why decode-focused techniques (paging, continuous batching, speculation, KV-quant) dominate the field, and it motivates structural fixes — chunked prefill (Sarathi, arXiv:2403.02310) and prefill/decode disaggregation (Splitwise arXiv:2311.18677, DistServe arXiv:2401.09670) that stop the two phases from contending for the same GPU.

Taught in

• 12-llm-inference-systems — §2 the two phases, §8 disaggregation/chunked prefill.
• 13-gpu-engineer-roadmap — §7 “design a serving stack” interview map.

See also

• kv-cache — populated by prefill, read by decode
• continuous-batching · speculative-decoding · quantization-inference
• paged-attention · arithmetic-intensity · roofline-model · flash-attention
• gpu-systems-for-llms