Learn: LLM Inference & Serving Systems

What you're learning

Why serving an LLM is a different systems problem from training it, and the techniques that make it fast and cheap: the prefill/decode split, the KV cache and why it dominates memory, PagedAttention, continuous batching, speculative decoding, and quantization. By the end you should be able to size a KV cache, explain why decode is memory-bound, and reason about the throughput/latency tradeoff of batching.

This is Tier 4, lesson 1. It assumes the memory hierarchy and roofline (02-gpu-memory-hierarchy, 03-performance-modeling-roofline) and attention/FlashAttention (06-memory-wall-and-flashattention).

Sourcing note

The web-search budget for this session was exhausted before this lesson was written, so the fast-moving framework details (exact current vLLM/SGLang/TensorRT-LLM behavior as of late 2026) could not be freshly re-verified. The algorithms below are cited to their primary papers and are stable; treat specific framework-version claims as [verify] and check the project docs before relying on them.

---

1. Learning map

graph TD
    A["Inference ≠ training"] --> B["Two phases:<br/>prefill vs decode"]
    B --> C["Prefill: compute-bound<br/>(big GEMM over the prompt)"]
    B --> D["Decode: memory-bound<br/>(1 token/step, autoregressive)"]
    D --> E["KV cache<br/>stores past K,V"]
    E --> F["KV cache memory<br/>often the bottleneck"]
    F --> G["PagedAttention / vLLM<br/>block-based KV mgmt"]
    D --> H["Continuous batching<br/>iteration-level scheduling"]
    D --> I["Speculative decoding<br/>draft + verify"]
    F --> J["Quantization<br/>weights / activations / KV"]
    C --> K["Disaggregation &<br/>chunked prefill"]
    D --> K
    G --> L["High-throughput serving"]
    H --> L
    I --> L
    J --> L
    K --> L

    style B fill:#44a,color:#fff
    style D fill:#a44,color:#fff
    style G fill:#4a4,color:#fff

---

2. Why inference is a different problem

Training is one giant, compute-bound, throughput-oriented job: you control the batch, you saturate the tensor cores, you measure MFU. Serving is the opposite in almost every dimension — it is latency-sensitive, bursty, autoregressive, and memory-bound, and the workload is split into two phases with opposite performance profiles.

Prefill processes the entire prompt at once. All $s_{\text{prompt}}$ tokens go through the model in parallel, so the attention and MLP matmuls are large GEMMs — compute-bound, high arithmetic intensity (lesson 03). Prefill produces the first output token and populates the KV cache.

Decode generates the rest, one token at a time. Each step feeds a single new token through the model, so every weight matrix is multiplied by an activation of batch-size-many rows — a skinny GEMM / GEMV. Arithmetic intensity is tiny; you re-read the entire model’s weights from HBM to produce one token per sequence. Decode is memory-bandwidth-bound, and because generation is autoregressive it cannot be parallelized across time within a sequence.

The asymmetry that defines serving

Prefill is compute-bound and happens once; decode is memory-bound and happens once per output token. A 500-token response is ~1 prefill + ~500 decode steps, so decode dominates cost and latency. Almost every inference optimization is really “make decode cheaper” — and since decode is memory-bound, the lever is batching (amortize the weight reads across many sequences) and moving fewer bytes (quantization, KV-cache management).

Serving metrics reflect this split:

• TTFT (time to first token) — dominated by prefill; matters for perceived responsiveness.
• ITL / TPOT (inter-token latency / time per output token) — dominated by decode; sets the streaming speed.
• Throughput (tokens/s across all requests) — what you optimize for cost; traded against latency via batch size.

---

3. The KV cache

At decode step $t$, computing attention for the new token needs the keys and values of all previous tokens. Recomputing them every step would be $O(s^2)$ work; instead we cache them. That cache is the central data structure of LLM serving.

Memory cost

For a model with $L$ layers, $n_{kv}$ key/value heads (with grouped-query attention¹; $n_{kv}=n_{\text{heads}}$ for full MHA, $n_{kv}=1$ for MQA²), head dimension $d_h$, sequence length $s$, batch $b$, and $p$ bytes/element, the KV cache holds both K and V:

$M_{\text{KV}}=2\cdot L\cdot n_{kv}\cdot d_h\cdot s\cdot b\cdot p$

Worked: KV cache for a 70B-class model

Take Llama-3-70B-style dims: $L=80$, $n_{kv}=8$ (GQA), $d_h=128$, FP16 ($p=2$).
Per token, all layers: $2\cdot 80\cdot 8\cdot 128\cdot 2=327,680$ bytes $\approx 320$ KiB/token.
One 8192-token sequence: $320\text{ KiB}\times 8192\approx 2.5$ GiB.
Batch of 32 such sequences: $\approx 80$ GiB — an entire H100’s memory, on top of the 140 GB of FP16 weights. Without GQA ($n_{kv}=64$) it would be 8× worse. This is why KV-cache management, not FLOPs, is the binding constraint on serving throughput.

The takeaway: at serving batch sizes, the KV cache rivals or exceeds the weights in memory, and it grows with every concurrent sequence and every generated token. How you allocate it decides how many requests you can run at once — i.e., your throughput.

Quiz: Why is the decode phase memory-bound while prefill is compute-bound, for the same model?

Answer

Arithmetic intensity (lesson 03) = FLOPs / bytes moved. In prefill, each weight matrix is multiplied by an activation with $s_{\text{prompt}}$ rows, so the weight bytes (read once from HBM) are amortized over $s_{\text{prompt}}$ tokens of math → high intensity → compute-bound. In decode, each weight matrix is multiplied by an activation with only $b$ rows (one new token per sequence). You still read the entire weight matrix from HBM but do far less math with it, so intensity is $\sim b$ and tiny → you sit on the memory roof. That’s also why increasing the decode batch $b$ raises intensity and thus throughput — until the KV cache runs out of memory.

---

4. PagedAttention and vLLM

Naively, you’d allocate each sequence a contiguous KV buffer sized to the max context. This is disastrous: you don’t know the output length in advance, so you either over-allocate (massive internal fragmentation — reserving 8k slots for a 200-token reply) or you can’t grow. Classic serving systems wasted 60–80% of KV memory this way.³

PagedAttention (Kwon et al., SOSP 2023 — the algorithm behind vLLM) borrows virtual memory paging.³ The KV cache is divided into fixed-size blocks (e.g. 16 tokens each). A sequence’s logical tokens map to physical blocks through a per-sequence block table; blocks need not be contiguous and are allocated on demand as the sequence grows. Results:

• Near-zero fragmentation — waste is bounded to at most one partial block per sequence (< 4%).
• Sharing — a shared prompt prefix (system prompt, few-shot examples) or the branches of beam/parallel sampling can point at the same physical blocks, with copy-on-write when they diverge. This slashes memory for high-fan-out sampling.
• Higher throughput — more sequences fit in memory → larger decode batch → higher intensity → more tokens/s. vLLM reported multi-fold throughput gains over prior serving systems at the same latency.³

graph LR
    subgraph "Logical view (per sequence)"
      T["tokens 0..N"]
    end
    subgraph "Block table"
      BT["logical block → physical block"]
    end
    subgraph "Physical KV blocks (HBM, non-contiguous)"
      P0["blk 7"]; P1["blk 3"]; P2["blk 9"]; P3["blk 1"]
    end
    T --> BT --> P0 & P1 & P2 & P3

PagedAttention: a sequence’s tokens are scattered across non-contiguous fixed-size physical blocks via a block table — like OS virtual memory. Shared prefixes reference the same physical blocks (copy-on-write on divergence).

The attention kernel is modified to gather K/V from the paged (non-contiguous) blocks rather than a flat buffer — the systems analogue of a fused, gather-aware FlashAttention (lesson 06).

Quiz: A serving system pre-allocates a contiguous KV buffer sized to the max context for every request. Name the two waste modes, and how paging fixes each.

Answer

(1) Internal fragmentation: a request that ends up generating 200 tokens still reserved (say) 8192 slots — the unused 7992 slots are stranded for that sequence’s whole lifetime. (2) Reserved-but-unused / external fragmentation: because buffers are large and contiguous, free memory gets chopped into holes too small to fit a new max-length buffer, so you can’t admit requests even though total free memory suffices. Paging fixes both: KV is allocated in small fixed blocks on demand, so a sequence only ever holds blocks it’s actually using (waste ≤ one partial block, <4%), and any free block can serve any sequence (no contiguity requirement → no external fragmentation). More live sequences fit → larger decode batch → higher throughput.

Quiz: 4-bit weight-only quantization gives a large speedup at batch size 1 but little or none at batch size 64. Why?

Answer

At batch 1, decode is deeply memory-bound: the runtime is dominated by reading weights from HBM, so cutting weights 16→4 bits (~4× fewer bytes) nearly directly speeds it up. As batch grows, arithmetic intensity rises (the same weight read is amortized over more sequences), pushing decode toward compute-bound; now the binding cost is the matmul, and weight-only 4-bit adds on-the-fly dequant work without speeding the FP16 math — so the win shrinks or reverses. This is the roofline (lesson 03) again: weight-only quant helps in the memory-bound regime; in the compute-bound regime you need a scheme that speeds the matmul itself (W8A8 / FP8 on tensor cores).

---

5. Continuous (iteration-level) batching

With static batching, you group $B$ requests, run them together, and can’t return the batch until the longest sequence finishes — short replies wait idle behind long ones, and new arrivals wait for the whole batch. Because response lengths vary wildly, GPU utilization craters.

Continuous batching (a.k.a. iteration-level scheduling; introduced by Orca, OSDI 2022⁴) schedules at the granularity of a single decode step, not a whole request. After every token, the scheduler can evict finished sequences and admit waiting ones into the running batch. The GPU is kept full of active sequences at all times. Combined with PagedAttention (which makes admitting/evicting cheap in memory), this is the backbone of modern high-throughput serving.

Why paging and continuous batching are a package deal

Continuous batching constantly adds and removes sequences of unpredictable length. That only works if KV memory can be allocated and freed in small increments without fragmentation — which is exactly what PagedAttention provides. Together they turn “GPU idle behind the longest request” into “GPU always running a full batch of live tokens.”

---

6. Speculative decoding

Decode is memory-bound: each step reads all weights to produce one token, so the ALUs are mostly idle. Speculative decoding⁵⁶ exploits that slack. A small, cheap draft model proposes $k$ tokens autoregressively; the large target model then verifies all $k$ in a single forward pass (parallel over the $k$ positions — a prefill-like, higher-intensity step). A carefully designed acceptance test accepts the longest correct prefix and provably preserves the target model’s output distribution — it is exact, not an approximation.⁵

• If the draft is accurate, you get several tokens per target forward pass → speedup roughly proportional to the mean accepted length; typical wins are ~2–3× with negligible quality change (workload- and draft-dependent).
• The speedup hinges on the acceptance rate and the draft’s cost; a bad draft can even slow you down.

Variants remove the need for a separate draft model:

• Medusa⁷ adds extra decoding “heads” to the target model to predict several future tokens at once, verified with a tree-attention scheme.
• EAGLE / EAGLE-2⁸ draft at the feature (hidden-state) level rather than the token level, achieving higher acceptance and stronger speedups; EAGLE-family methods are among the strongest reported as of this writing ([verify] current SOTA).

Quiz: Speculative decoding does more total FLOPs than plain decoding (draft + verify). Why is it faster?

Answer

Because decode is memory-bound, not compute-bound. Plain decode reads all target weights from HBM to emit one token — the tensor cores sit idle. Speculative decoding turns one memory-bound step into a compute-heavier but batched verification of $k$ candidate tokens in a single target pass: it reads the target weights once but produces up to $k$ tokens. The extra FLOPs (draft model + verifying rejected tokens) are nearly free because they run in the compute headroom that decode was wasting. You trade abundant idle FLOPs for scarce HBM bandwidth. If decode were compute-bound, speculation would not help.

---

7. Quantization for inference

Since decode is bottlenecked on moving weights (and KV) through HBM, storing them in fewer bits directly buys bandwidth and capacity. Inference quantization comes in three flavors:

• Weight-only (W8/W4, activations stay FP16): GPTQ⁹ (second-order, one-shot post-training) and AWQ¹⁰ (activation-aware; protects the salient weight channels). 4-bit weights ~4× shrink the weights with minimal accuracy loss; ideal for memory-bound decode and single-stream latency. The matmul dequantizes on the fly.
• Weight + activation (W8A8 / FP8): SmoothQuant¹¹ migrates activation outliers into the weights so both can be INT8; FP8 (E4M3)¹² is now hardware-accelerated on Hopper/Blackwell tensor cores, giving compute and bandwidth wins — increasingly the default for high-throughput serving.
• KV-cache quantization (KV8/KV4): quantize the cache itself to fit more/longer sequences. Directly attacks the §3 bottleneck; some accuracy risk on long context.

Match the quantization to the bottleneck

Weight-only 4-bit helps latency-bound, small-batch decode (bandwidth is the wall) but does little for compute-bound prefill or large-batch serving, where the dequant overhead can even hurt — there, W8A8/FP8 that actually speeds the matmul is the win. “Which precision” is a roofline question: know whether you’re memory- or compute-bound first (lesson 03).

---

8. Disaggregation and chunked prefill

Prefill (compute-bound, bursty) and decode (memory-bound, steady) contend for the same GPU. Two structural fixes:

• Chunked prefill (Sarathi / Sarathi-Serve¹³): split a long prompt’s prefill into chunks and interleave them with ongoing decode steps in the same batch, so a big prefill doesn’t stall everyone’s token stream (bounding ITL) while still keeping the GPU busy.
• Prefill/decode disaggregation (Splitwise¹⁴, DistServe¹⁵): run prefill on one pool of GPUs and decode on another, transferring the KV cache between them. Each pool is tuned for its phase (and even its hardware), and the two stop interfering — improving goodput under latency SLOs. Now common in large-scale serving stacks ([verify] exact current designs).

---

9. GPU/LLM connection

Inference optimization is a large fraction of frontier-lab GPU engineering — it’s where most served compute goes, and small percentage wins translate to enormous cost savings at scale. Every technique here is an application of the earlier tiers:

• decode being memory-bound is a roofline statement (lesson 03);
• PagedAttention is memory-hierarchy-aware allocation (lesson 02) plus a gather-aware FlashAttention kernel (lesson 06);
• quantization is precision/bandwidth engineering (lesson 03);
• disaggregation is a parallelism/placement decision cousin to lessons 08–11.

---

10. Practice & resources

Problem: size the serving batch

A model has FP16 weights of 140 GB served on 2×H100 (160 GB total). Its KV cache is 320 KiB/token (§3). You want to serve requests averaging 4k context. Roughly how many concurrent sequences fit, and what limits it?

Solution. Weights take 140 GB, leaving ~20 GB for KV (plus activations/overhead — call it ~16 GB usable). Per sequence at 4k tokens: $320\text{ KiB}\times 4096\approx 1.25$ GiB. So $\sim 16/1.25\approx 12$–13 concurrent sequences. The limit is KV-cache capacity, not compute — which is exactly why PagedAttention (pack more sequences in), KV quantization (halve per-token cost), and GQA (fewer KV heads) all directly raise serving throughput. Tensor parallelism across the 2 GPUs (lesson 10) also frees weight memory per GPU for more KV.

Hands-on:

• Serve a model with vLLM (vllm serve <model>); hit it with a load generator and measure TTFT, TPOT, and throughput as you vary concurrency. Watch throughput rise then latency spike as the KV cache saturates.
• Compare an FP16 vs a 4-bit (AWQ/GPTQ) checkpoint for single-stream latency vs batched throughput — see the §7 tradeoff empirically.
• Toggle a serving engine’s speculative decoding and measure the acceptance rate and speedup on your workload.

Resources:

• PagedAttention / vLLM paper³ and the vLLM docs.
• Orca (continuous batching)⁴; Sarathi-Serve (chunked prefill)¹³.
• Speculative decoding⁵⁶, Medusa⁷, EAGLE⁸.
• Quantization: GPTQ⁹, AWQ¹⁰, SmoothQuant¹¹, FP8 formats¹².
• Disaggregation: Splitwise¹⁴, DistServe¹⁵.

---

11. What’s next

• 13-gpu-engineer-roadmap — the capstone: turn all of this into portfolio projects, practice, and interview readiness.
• Back to 11-sota-large-scale-training (the training-side counterpart) and the hub gpu-systems-for-llms.

Related concepts: kv-cache, paged-attention, continuous-batching, speculative-decoding, quantization-inference, prefill-decode, and the foundations arithmetic-intensity, roofline-model, flash-attention.

---

References

Reference concepts: kv-cache, paged-attention, continuous-batching, speculative-decoding, quantization-inference | Topic: gpu-systems-for-llms | Filed: 2026-09-02

Footnotes

1. Ainslie et al., “GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints” — arXiv:2305.13245 (2023). Grouped-query attention: $n_{kv}$ between 1 (MQA) and $n_{\text{heads}}$ (MHA), the standard KV-cache reduction in modern LLMs. ↩

2. Shazeer, “Fast Transformer Decoding: One Write-Head is All You Need” — arXiv:1911.02150 (2019). Multi-query attention ($n_{kv}=1$). ↩

3. Kwon et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention” — arXiv:2309.06180, SOSP 2023. The vLLM system; source of the fragmentation-waste and throughput claims. [established] ↩ ↩² ↩³ ↩⁴

4. Yu et al., “Orca: A Distributed Serving System for Transformer-Based Generative Models” — OSDI 2022. Introduced iteration-level (continuous) batching. [established] ↩ ↩²

5. Leviathan et al., “Fast Inference from Transformers via Speculative Decoding” — arXiv:2211.17192, ICML 2023. The draft-and-verify algorithm with distribution-preserving acceptance. [established] ↩ ↩² ↩³

6. Chen et al., “Accelerating Large Language Model Decoding with Speculative Sampling” — arXiv:2302.01318 (2023). Concurrent formulation. ↩ ↩²

7. Cai et al., “Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads” — arXiv:2401.10774 (2024). ↩ ↩²

8. Li et al., “EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty” — arXiv:2401.15077 (2024); EAGLE-2 arXiv:2406.16858. Feature-level drafting. [recent — verify current SOTA] ↩ ↩²

9. Frantar et al., “GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers” — arXiv:2210.17323 (2022). ↩ ↩²

10. Lin et al., “AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration” — arXiv:2306.00978, MLSys 2024. ↩ ↩²

11. Xiao et al., “SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models” — arXiv:2211.10438, ICML 2023. ↩ ↩²

12. Micikevicius et al., “FP8 Formats for Deep Learning” — arXiv:2209.05433 (2022). E4M3 / E5M2. ↩ ↩²

13. Agrawal et al., “SARATHI” (arXiv:2308.16369) and “Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve” — arXiv:2403.02310, OSDI 2024. Chunked prefill. [established] ↩ ↩²

14. Patel et al., “Splitwise: Efficient Generative LLM Inference Using Phase Splitting” — arXiv:2311.18677 (2023). Prefill/decode disaggregation. ↩ ↩²

15. Zhong et al., “DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving” — arXiv:2401.09670, OSDI 2024. ↩ ↩²