Learn: MQA → GQA → MLA (Shrinking the KV Cache)

What you're learning

At inference, the KV cache — not FLOPs — is usually what caps how many sequences you can serve and how long a context you can hold. This lesson walks the arc of KV-cache reduction: full multi-head attention (MHA) → MQA (one KV head) → GQA (a few KV groups, the current production default) → MLA (a low-rank latent cache, DeepSeek’s frontier trick). By the end you should be able to write each scheme’s per-token cache formula from memory, explain why MLA beats GQA on both memory and quality at once, and derive why MLA needs a separate “decoupled RoPE” key.

This lesson assumes lessons 01–04: scaled dot-product attention (01-scaled-dot-product-attention), the multi-head block (02-multi-head-attention-and-the-block), positional encodings including RoPE (03-positional-information), and the quadratic bottleneck / IO-aware view (04-quadratic-bottleneck-and-io-aware). It is the attention-side companion to the serving-systems view of the KV cache in 12-llm-inference-systems. Vocabulary follows the attention-glossary.

---

1. Learning map

graph TD
    A["Inference bottleneck:<br/>KV cache grows with<br/>tokens × heads × layers"] --> B["Lever: reduce the number<br/>of DISTINCT K/V heads stored"]
    B --> C["MHA<br/>n_kv = n_heads<br/>(baseline, biggest cache)"]
    C --> D["MQA<br/>n_kv = 1<br/>÷h cache, quality drops"]
    C --> E["GQA<br/>1 < g < h groups<br/>the memory/quality sweet spot"]
    D --> E
    E --> F["MLA<br/>cache a low-rank latent c_t<br/>reconstruct K,V per head"]
    F --> G["Weight absorption:<br/>fold up-projs into W^Q / W^O<br/>→ never materialize K,V"]
    F --> H["Decoupled RoPE:<br/>separate rope-carrying key<br/>(RoPE ⊥ absorption)"]
    G --> I["Smaller cache than GQA<br/>AND quality ≥ MHA"]
    H --> I

    style A fill:#a44,color:#fff
    style E fill:#44a,color:#fff
    style F fill:#4a4,color:#fff
    style I fill:#a83,color:#fff

---

2. Intuition: stop storing a full set of keys and values per head

Recall the serving picture from 12-llm-inference-systems: autoregressive decode generates one token at a time, and to attend to the past without recomputing it $O(s^2)$ times we cache every past token’s keys and values. That KV cache is

$M_{\text{KV}}=\underset{K\text{ and }V}{\underbrace{2}}\cdot L\cdot n_{kv}\cdot d_h\cdot s\cdot b\cdot p$

bytes, where $L$ is layers, $n_{kv}$ the number of key/value heads, $d_h$ the head dimension, $s$ the sequence length, $b$ the batch, and $p$ bytes per element. It grows with every concurrent sequence and every generated token — and at serving batch sizes it rivals or exceeds the model weights in HBM. That is the wall.

Stare at that formula and ask which factor you can actually move. $L$, $s$, $b$, $p$ are fixed by the model and the workload (well — $p$ you can shrink with KV quantization, a separate lever). The one architectural knob is $n_{kv}$: how many distinct key/value heads you store per layer.

Here is the key observation from lesson 02: in MHA every one of the $h$ query heads has its own K head and V head. But do the value/key subspaces really need to be that different across heads? Empirically, no — you can make many query heads share the same keys and values and lose surprisingly little. That single idea — decouple the number of query heads from the number of KV heads — generates the whole progression:

• MHA: $n_{kv}=h$. Every query head gets a private K/V head. Maximum expressivity, maximum cache.
• MQA: $n_{kv}=1$. All query heads share a single K/V head. Cache shrinks by a factor of $h$.
• GQA: $1<n_{kv}=g<h$. Query heads are partitioned into $g$ groups; each group shares one K/V head. A dial between the two extremes.
• MLA: don’t store per-head K/V at all — store a small shared latent $c_t$ per token and reconstruct each head’s K and V on the fly. Breaks out of the $n_{kv}\cdot d_h$ box entirely.

The one-sentence version

MQA/GQA shrink the cache by cutting the number of K/V heads ($n_{kv}$); MLA shrinks it by cutting the width of what’s cached — storing a low-rank latent instead of full-width K,V — which lets it go smaller than GQA and keep MHA-level quality.

This is a decode-memory optimization, not a FLOPs optimization

Sharing KV heads barely changes prefill FLOPs (the score and $A\cdot V$ matmuls still sum over the full model dimension $d$; see §5). What it slashes is KV-cache bytes and therefore the memory-bandwidth traffic of decode — which is exactly the binding constraint at serving time (lesson 12). Don’t expect MQA/GQA to speed up a compute-bound prefill much; expect it to let you fit more sequences and longer contexts, and to make each memory-bound decode step read fewer KV bytes.

Quiz (retrieval): Of the factors $2,L,n_{kv},d_h,s,b,p$ in the KV-cache formula, which one do MQA and GQA target, and why is it the natural lever?

Answer

They target $n_{kv}$, the number of distinct key/value heads. $L$, $d_h$, $s$, $b$ are dictated by the model architecture and the request; $p$ is a precision/quantization choice (orthogonal). $n_{kv}$ is the one term you can lower architecturally by letting multiple query heads share the same K and V — which is empirically cheap in quality. (MLA later attacks a different factor: it replaces the whole $n_{kv}\cdot d_h$ width with a small latent width $d_c$.)

---

3. The math

Throughout: $h$ = number of query heads, $d_h$ = head dimension, $d=h{d}_h$ = model dimension (assume $Q$/$K$/$V$ head dims equal for clarity), $L$ = layers.

3.1 MQA — one write-head is all you need

Multi-Query Attention¹ keeps all $h$ query heads but uses a single key head and a single value head shared across every query head. Concretely, $W^Q$ still produces $h$ query heads, but $W^K,W^V$ each produce only one $d_h$-wide head:

$q_t^{(i)}=W_i^Q{h}_t(i=\mathrm{1..}h),k_t=W^K{h}_t,v_t=W^V{h}_t(\text{shared}).$

Head $i$‘s attention is $\mathrm{softmax}(q_t^{(i)\top }{k}_{\le t}/\sqrt{d_h})v_{\le t}$ — every head reads the same cached $k,v$. The cache stores one K and one V per token per layer:

$M_{\text{KV}}^{\text{MQA}}=2L{d}_{h}sbp,\text{a factor of }h\text{ smaller than MHA.}$

The win is huge (for a 32-head model, $32\times$ less KV). The cost is quality: collapsing $h$ key/value subspaces into one removes representational capacity, and MQA-trained models show measurable degradation and can be training-unstable on large models.² That instability motivated the middle ground.

3.2 GQA — grouped query attention, the sweet spot

Grouped-Query Attention² interpolates: partition the $h$ query heads into $g$ groups ($1<g<h$, with $g\mid h$), and give each group its own shared K/V head. So $n_{kv}=g$:

$M_{\text{KV}}^{\text{GQA}}=2Lg{d}_{h}sbp\propto g.$

$g=h$ recovers MHA; $g=1$ recovers MQA. The GQA paper’s two findings are the reason it is everywhere:

1. Quality/memory sweet spot. A moderate $g$ (e.g. $g=8$) recovers essentially all of MHA’s quality while keeping most of MQA’s memory savings — the quality-vs-$g$ curve is steep near $g=1$ and flat by a handful of groups.²
2. Uptraining, not retraining. You can convert an existing MHA checkpoint to GQA by mean-pooling the K/V projections of each group and fine-tuning for a small fraction (~5%) of the original compute — no pretrain from scratch.²

This is why Llama-2-70B / Llama-3-class models and most production LLMs use GQA with $g\approx 8$: near-MHA quality at $\sim h/g$ less KV.²

Why grouping is stable where MQA is not

MQA forces all heads through a single 1-dimensional-per-token K/V bottleneck; the gradient signal to that lone head is an average over conflicting demands, which destabilizes large-model training. GQA gives each group its own K/V, so the shared head only has to serve a handful of similar query heads — enough capacity to stay stable and expressive while still cutting the cache several-fold.

Quiz (retrieval): GQA with $g=8$ on a 64-head model. By what factor is the KV cache smaller than MHA, and what is the cheap trick that lets you get a GQA model without pretraining from scratch?

Answer

Cache $\propto n_{kv}$, so $64/8=8\times$ smaller than MHA (and $8\times$ larger than MQA). The trick is uptraining: take a pretrained MHA checkpoint, mean-pool the $K$ (and $V$) projection matrices of the heads within each of the 8 groups into one shared head, then fine-tune briefly (~5% of pretraining compute) to recover quality.² No from-scratch pretrain needed.

3.3 MLA — Multi-head Latent Attention

GQA still stores full-width K and V, just for fewer heads. Multi-head Latent Attention (MLA, DeepSeek-V2)³ asks: what if we cache neither — and instead store a single low-rank latent per token, reconstructing per-head K and V only at the moment of use?

Down-projection (the only thing cached). From the token’s hidden state $h_t$, compute a compressed KV latent

$c_t^{KV}=W^{DKV}{h}_t\in {\mathbb{R}}^{d_c},d_c\ll h{d}_h.$

This $d_c$-wide vector $c_t^{KV}$ is what goes in the cache — one per token per layer, shared across all heads.

Up-projection (reconstruct at use time). Per-head keys and values are recovered by up-projecting the latent:

$k_t^{C,(i)}=W_i^{UK}{c}_t^{KV},v_t^{C,(i)}=W_i^{UV}{c}_t^{KV}.$

Queries get their own (optional) low-rank compression $c_t^Q=W^{DQ}{h}_t,q_t^{C,(i)}=W_i^{UQ}{c}_t^Q$, which shrinks activation memory and parameters during training but doesn’t affect the cache.

Naively this looks worse — you’d reconstruct full K,V for every head every step, materializing exactly what MHA stored. The magic is that you never have to.

Weight absorption — never materialize K, V

The compressed part of the attention score for query head $i$ is

$q_t^{C,(i)\top }{k}_s^{C,(i)}=\left(W_i^{UQ}{c}_t^Q{)}^{\top }\right(W_i^{UK}{c}_s^{KV})=c_t^{Q\top }\underset{\text{absorb into one matrix}}{\underbrace{\left(W_i^{UQ\top }{W}_i^{UK}\right)}}{c}_s^{KV}.$

Because $W_i^{UQ}$ and $W_i^{UK}$ are fixed weights, their product ${\tilde{W}}_i=W_i^{UQ\top }{W}_i^{UK}$ can be precomputed and folded into the query path. At inference you compute an effective query ${\tilde{q}}_t={\tilde{W}}_i^{\top }{c}_t^Q$ and dot it directly against the cached latent $c_s^{KV}$ — you never reconstruct $k_s^{C,(i)}$. Symmetrically, $W_i^{UV}$ absorbs into the output projection $W^O$, so you never materialize $v_s^{C,(i)}$ either: the value read is a projection of the same cached latent. Net effect: you cache only $c_s^{KV}$ (width $d_c$), yet compute exactly MHA-style multi-head attention.

Decoupled RoPE — why a separate key part is mandatory

There is a catch, and it is the subtle heart of MLA. The absorption above works only because ${\tilde{W}}_i=W_i^{UQ\top }{W}_i^{UK}$ is a single constant matrix. RoPE breaks that. RoPE (03-positional-information) multiplies each key by a position-dependent rotation $R_s$ (and each query by $R_t$), so a RoPE’d score term would be

$\left(R_t{W}_i^{UQ}{c}_t^Q{)}^{\top }\right(R_s{W}_i^{UK}{c}_s^{KV})=c_t^{Q\top }{W}_i^{UQ\top }\underset{=R_{s-t}}{\underbrace{R_t^{\top }{R}_s}}{W}_i^{UK}{c}_s^{KV}.$

Now a position-dependent factor $R_{s-t}$ sits between $W_i^{UQ}$ and $W_i^{UK}$. It does not commute with them, so you can no longer collapse them into one constant matrix — the “absorbed” weight would depend on the token pair $(t,s)$, defeating the whole scheme (you’d be back to materializing per-head, per-position keys). RoPE and weight absorption are fundamentally incompatible.

DeepSeek’s fix is decoupled RoPE: carry positional information on a separate, small key/query part that lives outside the compressed path.

• A shared rope-carrying key $k_t^R=R_t(W^{KR}{h}_t)\in {\mathbb{R}}^{d_h^R}$, generated straight from $h_t$ (one head, MQA-style — shared across all heads), and cached alongside the latent.
• A per-head rope query $q_t^{R,(i)}=R_t(W_i^{QR}{c}_t^Q)\in {\mathbb{R}}^{d_h^R}$.

The final query and key for each head are the concatenation of the compressed (position-free, “NoPE”) part and the decoupled RoPE part, so the score splits into two additive terms:

$q_t^{(i)\top }{k}_s^{(i)}=\underset{\text{absorbable, no position}}{\underbrace{q_t^{C,(i)\top }{k}_s^{C,(i)}}}+\underset{\text{carries RoPE, tiny width }{d}_h^R}{\underbrace{q_t^{R,(i)\top }{k}_s^R}}.$

The compressed term stays fully absorbable (no RoPE inside it); the RoPE term is small and shared. So the total cache per token per layer is $d_c+d_h^R$ elements — the latent plus the shared rope key.

$\boxed{M_{\text{KV}}^{\text{MLA}}=L(d_c+d_h^R)sbp}$

(Often quoted as $\approx L{d}_{c}sbp$ since $d_h^R\ll d_c$.)

The numbers. DeepSeek-V2³ uses $d_h=128$, $d_c=4d_h=512$, and a decoupled RoPE width $d_h^R=d_h/2=64$, so it caches $d_c+d_h^R=576$ elements/token/layer. Compare full MHA at those dims ($n_h=128$ heads): $2n_h{d}_h=2\cdot 128\cdot 128=32768$ elements — MLA is ~57× smaller. DeepSeek report this equals the cache of GQA with just $g=2.25$ groups (since $2g{d}_h=576\Rightarrow g=2.25$), while matching or exceeding full MHA quality.³ That is the punchline: MLA lands below the GQA cache curve and above the GQA quality curve — it dominates GQA on both axes. DeepSeek-V3 carries MLA forward at larger scale.⁴

MLA is not "just low-rank KV"

Two things make it work and are easy to miss. (1) Weight absorption — without folding the up-projections into $W^Q$/$W^O$, you’d reconstruct full-width per-head K,V at every step and gain nothing (you’d cache small but compute like MHA with extra up-proj FLOPs). (2) Decoupled RoPE — you cannot just RoPE the reconstructed keys, because the position rotation sits between the two absorbable matrices and kills the absorption. The separate rope key is not a detail; it is what makes MLA + RoPE compatible at all.

Quiz (retrieval): Why does MLA beat GQA on both memory and quality, whereas MQA/GQA trade quality for memory?

Answer

MQA/GQA reduce the cache by discarding representational capacity — they force many query heads to share one full-width K/V head, so fewer distinct key/value subspaces exist; you buy memory with quality. MLA reduces the cache differently: it caches a low-rank latent $c_t$ (width $d_c+d_h^R$) but, via weight absorption, still computes full multi-head attention with per-head K/V reconstructed from that latent — so it keeps (empirically, even improves on³) MHA-level expressivity. The reduction comes from the K/V matrices being genuinely low-rank, not from throwing heads away. Result: cache smaller than GQA-2.25 and quality ≥ MHA — a strict improvement on GQA’s frontier, not a point on its tradeoff curve.

Quiz (retrieval): MLA needs a separate "decoupled RoPE" key of width $d_h^R$. State precisely why RoPE cannot just be applied to the reconstructed keys.

Answer

Weight absorption relies on collapsing $W_i^{UQ\top }{W}_i^{UK}$ into a single position-independent matrix so the query can be dotted straight against the cached latent $c_s^{KV}$. Applying RoPE inserts a position-dependent rotation $R_{s-t}$ between $W_i^{UQ}$ and $W_i^{UK}$ in the score $c_t^{Q\top }{W}_i^{UQ\top }{R}_{s-t}{W}_i^{UK}{c}_s^{KV}$. $R_{s-t}$ does not commute with those weight matrices, so the “absorbed” matrix would depend on the token pair $(t,s)$ — you could no longer precompute it, and you’d be forced to materialize per-head, per-position keys (losing the whole benefit). The fix: put RoPE on a separate small key part $k^R$ (shared across heads, width $d_h^R$) that lives outside the absorbable compressed path, and add its score term separately. Compressed part stays absorbable; RoPE part carries position.

---

4. Illustration: what each scheme stores per token

graph TB
    subgraph MHA["MHA — n_kv = h (baseline)"]
        direction LR
        m1["K₁ V₁"]; m2["K₂ V₂"]; m3["K₃ V₃"]; mdots["…"]; mh["K_h V_h"]
    end
    subgraph GQA["GQA — n_kv = g groups"]
        direction LR
        g1["K₁ V₁<br/>(heads 1..h/g)"]; g2["K₂ V₂<br/>(next group)"]; gdots["…"]; gg["K_g V_g"]
    end
    subgraph MQA["MQA — n_kv = 1"]
        direction LR
        q1["K V<br/>(shared by ALL heads)"]
    end
    subgraph MLA["MLA — cache latent c_t + shared k^R"]
        direction LR
        l1["c_t (d_c)"]; l2["k^R (d_h^R, shared)"]
        l1 -. "up-proj W^UK,W^UV<br/>(absorbed into W^Q,W^O)" .-> l3["per-head K,V<br/>materialized only transiently"]
    end

    MHA --> GQA --> MQA
    MHA -. "low-rank, not fewer heads" .-> MLA

    style MHA fill:#a44,color:#fff
    style GQA fill:#44a,color:#fff
    style MQA fill:#864,color:#fff
    style MLA fill:#4a4,color:#fff

Reading the figure: MHA→GQA→MQA all store full-width K,V, just for fewer heads (boxes get fewer, each box unchanged). MLA is different in kind — it stores a narrow latent $c_t$ (plus a tiny shared rope key $k^R$) and reconstructs the per-head K,V transiently, which is why it can undercut even MQA’s width while keeping every head distinct.

Cache per token per layer, at a glance:

Scheme	Distinct K/V heads $n_{kv}$	Cached elements / token / layer	vs MHA
MHA	$h$	$2h{d}_h$	$1\times$
GQA	$g$	$2g{d}_h$	$g/h$
MQA	$1$	$2d_h$	$1/h$
MLA	— (latent)	$d_c+d_h^R$	$\frac{d_c+d_h^R}{2h{d}_h}$

---

5. FLOPs & memory

KV cache (bytes) — the number that actually binds serving. With $p$ bytes/element:

$$M_{\text{KV}}^{\text{MHA}}=2Lh{d}_{h}sbp,M_{\text{KV}}^{\text{GQA}}=2Lg{d}_{h}sbp,M_{\text{KV}}^{\text{MQA}}=2L{d}_{h}sbp,M_{\text{KV}}^{\text{MLA}}=L(d_c+d_h^R)sbp.$$

Note the general pattern $2L{n}_{kv}{d}_h$ for the head-sharing family vs MLA’s $L{d}_c$ (the factor of 2 disappears because K and V are both reconstructed from the one cached latent).

FLOPs. The head-sharing schemes barely change attention compute. The score $Q{K}^{\top }$ and $A\cdot V$ matmuls sum over the full model width regardless of $n_{kv}$ (the shared K/V head is broadcast to all query heads), so prefill FLOPs are ~unchanged; only the tiny K/V projection GEMMs shrink (from $2dd$ to $2dg{d}_h$ each). MLA adds up-projection work, but weight absorption folds it into the existing $W^Q$/$W^O$ GEMMs, so its inference FLOPs stay comparable while its bytes moved during decode drop sharply — and decode is memory-bound (lesson 12), so fewer KV bytes is the win. The upshot across all four: the lever is memory/bandwidth, not FLOPs.

Play with it

The Attention FLOPs & KV-cache calculator has an MHA / GQA / MQA / MLA comparison built in (the “KV cache: MHA vs GQA vs MQA vs MLA” table, with an MLA latent d_c field). Set a model’s $d$, $h$, $L$, $s$, pick $g$ for GQA and $d_c$ for MLA, and watch the four cache sizes and their ratios-to-MHA update live. Try $d=5120$, $h=128$, $g=8$, $d_c=512$ and confirm MLA lands below GQA.

Quiz (retrieval): You keep a model's architecture fixed but switch MHA → GQA( $g=8$). Which of {prefill FLOPs, decode-step bytes moved from the KV cache, KV-cache capacity} change substantially, and which barely move?

Answer

Prefill FLOPs barely move — the $Q{K}^{\top }$ and $A\cdot V$ matmuls still run over all $h$ query heads / full width $d$; only the small K,V projection GEMMs shrink. KV-cache capacity and per-decode-step KV bytes both drop ~$h/8$× — you store and re-read $8$ K/V heads instead of $h$. Since decode is memory-bandwidth-bound, that byte reduction is what raises serving throughput and max context, even though FLOPs are ~flat. (This is the whole reason head-sharing exists: it’s a memory optimization masquerading as an architecture change.)

---

6. Practice

Problem 1: MHA vs GQA vs MLA cache for a 70B-class model

Config: $L=80$ layers, $h=64$ query heads, $d_h=128$, FP16 ($p=2$ bytes). Compute the KV cache per token (all layers) for (a) MHA, (b) GQA with $g=8$, (c) MLA with $d_c=512$, $d_h^R=64$. Then give the size of one $s=8192$-token sequence under each.

Solution. Per-token-per-layer element counts × $2Lp$ (or $Lp$ for MLA):

• (a) MHA: $2Lh{d}_{h}p=2\cdot 80\cdot 64\cdot 128\cdot 2=2,621,440$ B $\approx 2.5MiB/token$. One 8192-seq: $\times 8192\approx 20\mathbf{GiB}$.
• (b) GQA (g=8): $8/64=1/8$ of MHA $=327,680$ B $\approx 320KiB/token$; 8192-seq $\approx 2.5\mathbf{GiB}$. (Matches the Llama-3-70B figure in lesson 12.)
• (c) MLA: $L(d_c+d_h^R)p=80\cdot 576\cdot 2=92,160$ B $\approx 90KiB/token$; 8192-seq $\approx 720\mathbf{MiB}$.
  MLA is ~28× smaller than MHA and ~3.6× smaller than GQA-8 here — and it does so at MHA-level quality. That extra headroom is directly more concurrent sequences / longer context per GPU (lesson 12’s batch-sizing arithmetic).

Problem 2: what GQA is MLA "equivalent to"?

For DeepSeek-V2’s dims ($d_h=128$, $d_c=512$, $d_h^R=64$), find the number of GQA groups $g$ that would give the same cache size as MLA. Then say why MLA is still the better choice at that equivalent point.

Solution. Set GQA cache = MLA cache per token per layer: $2g{d}_h=d_c+d_h^R\Rightarrow 2\cdot 128\cdot g=512+64=576\Rightarrow g=576/256=2.25$. So MLA costs the same KV as GQA with ~2.25 groups.³ But GQA at $g\approx 2$ sits in the steep part of the quality curve — that few groups noticeably hurts quality (close to MQA’s regime). MLA at the same memory keeps full multi-head expressivity (all $h$ heads distinct, K/V reconstructed from the latent), so it retains MHA-level quality that GQA-2 cannot. Same bytes, strictly more quality — MLA dominates that point of GQA’s tradeoff curve.

---

Read next / primary source

Read the three primary sources in order of the arc: Shazeer (MQA), “Fast Transformer Decoding: One Write-Head is All You Need”¹; Ainslie et al. (GQA)² — §2–3 for the uptraining recipe and the quality/$g$ ablation; DeepSeek-V2³ — §2.1 for MLA, weight absorption, and decoupled RoPE (the crux). DeepSeek-V3⁴ shows MLA at larger scale ([verify] exact config details against the report). For how these cache reductions cash out in a serving stack (PagedAttention, batch sizing, KV quantization), see 12-llm-inference-systems.

Ask your teacher

Fuzzy on the weight-absorption algebra, the non-commutativity of RoPE with absorption, or how to reason about the cache/quality frontier for a specific model? Ask — I’m your teacher for this course and can rederive any step, sketch the absorbed-matrix shapes explicitly, or build you a harder cache-sizing problem.

What’s next

• 06-sparse-and-sliding-window — the other axis of attention scaling: instead of shrinking the per-token cache, shrink how many tokens each query attends to (sliding-window, sparse, and hybrid patterns).
• Back to 04-quadratic-bottleneck-and-io-aware — the quadratic-compute and IO-aware view that motivates why we care about the memory side at all.

---

References

Reference: attention-glossary | Topic: attention-mechanisms | Filed: 2026-09-10

Footnotes

1. Shazeer, “Fast Transformer Decoding: One Write-Head is All You Need” — arXiv:1911.02150 (2019). Introduces multi-query attention ($n_{kv}=1$): all query heads share a single K/V head; ~$h\times$ smaller KV cache, with a quality/stability cost on large models. [established] ↩ ↩²

2. Ainslie et al., “GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints” — arXiv:2305.13245 (2023). Grouped-query attention ($1<g<h$), the quality/memory sweet spot; the mean-pool + ~5%-compute uptraining recipe to convert MHA checkpoints; the quality-vs-$g$ ablation. The standard KV reduction in Llama-2/3-class models. [established] ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷

3. DeepSeek-AI, “DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model” — arXiv:2405.04434 (2024). Introduces Multi-head Latent Attention (MLA): low-rank joint KV compression to a cached latent $c_t$, weight absorption of up-projections, and decoupled RoPE. Reports cache equal to GQA-2.25 while matching/exceeding MHA quality ($d_h=128$, $d_c=512$, $d_h^R=64$). [recent] ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶

4. DeepSeek-AI, “DeepSeek-V3 Technical Report” — arXiv:2412.19437 (2024). Carries MLA forward at larger scale (alongside DeepSeekMoE). [verify] exact MLA dimensions against the report before citing specific numbers. ↩ ↩²