KV Cache

Definition

The KV cache is the per-sequence store of already-computed key and value tensors for every past token, kept in HBM so autoregressive decode does not recompute them. At decode step $t$ the new token attends to all previous tokens’ K/V; caching turns the $O(s^2)$ recompute into $O(s)$ reads. It is the central data structure of LLM serving and, at realistic batch sizes, the binding memory constraint — not weights, not FLOPs.

Key math / How it works

For $L$ layers, $n_{kv}$ key/value heads, head dim $d_h$, sequence length $s$, batch $b$, and $p$ bytes/element, the cache holds both K and V:

$M_{\text{KV}}=2\cdot L\cdot n_{kv}\cdot d_h\cdot s\cdot b\cdot p$

Worked (Llama-3-70B-style): $L=80$, $n_{kv}=8$ (GQA), $d_h=128$, FP16 ($p=2$).

• Per token, all layers: $2\cdot 80\cdot 8\cdot 128\cdot 2=327,680$ B $\approx 320$ KiB/token.
• One 8192-token sequence: $320\text{ KiB}\times 8192\approx 2.5$ GiB.
• Batch of 32: $\approx 80$ GiB — an entire H100, on top of 140 GB of FP16 weights. Full MHA ($n_{kv}=64$) would be 8× worse.

GQA/MQA reduction. The formula scales linearly in $n_{kv}$, so the cheapest lever is fewer KV heads. MQA ($n_{kv}=1$, arXiv:1911.02150) shares one K/V head across all query heads; GQA ($1<n_{kv}<n_{\text{heads}}$, arXiv:2305.13245) interpolates, e.g. 64 query heads → 8 KV groups = 8× smaller cache with negligible quality loss. This is why every modern LLM ships GQA.

Why it matters

Decode reads the whole cache every step, so cache size sets both the memory ceiling (how many concurrent sequences fit → decode batch → throughput) and a bandwidth floor on decode latency. It grows with every concurrent sequence and every generated token, so managing it well (paged-attention, KV quantization, GQA) is the highest-leverage serving optimization. See the roofline: decode’s arithmetic intensity $\sim b$ is tiny, so the cache — not compute — is the wall.

Taught in

• 12-llm-inference-systems — §3 the memory formula, worked sizing, and why decode is memory-bound.

See also

• paged-attention — block-based allocation of this cache
• continuous-batching — admits/evicts sequences that own cache blocks
• prefill-decode — prefill populates it, decode reads it
• quantization-inference — KV-quant shrinks it directly
• flash-attention · arithmetic-intensity · roofline-model · gpu-memory-hierarchy
• gpu-systems-for-llms