MLA (Multi-head Latent Attention)

Definition

Multi-head Latent Attention (DeepSeek-V2, 2024; arXiv:2405.04434) shrinks the KV cache by caching neither per-head keys nor values, but a single low-rank latent $c_{t}$ per token, from which per-head K and V are reconstructed at use time. Via weight absorption the reconstruction is folded into the query/output projections so full-width K,V are never materialized, and a decoupled RoPE key carries position separately. It undercuts GQA on cache size and matches/exceeds MHA quality — dominating GQA on both axes.

Key math / How it works

Down-projection (the only thing cached). $c_{t}^{K V} = W^{D K V} h_{t} \in R^{d_{c}}$ , $d_{c} ≪ h d_{h}$ , shared across heads. Per-head K,V are recovered as $k_{t}^{C, (i)} = W_{i}^{U K} c_{t}^{K V}$ , $v_{t}^{C, (i)} = W_{i}^{U V} c_{t}^{K V}$ .

Weight absorption — never materialize K,V. The compressed score term is

$q_{t}^{C, (i) ⊤} k_{s}^{C, (i)} = c_{t}^{Q ⊤} \tilde{W}_{i}, constant (W_{i}^{U Q ⊤} W_{i}^{U K}) c_{s}^{K V},$

so $\tilde{W}_{i}$ is precomputed into the query path and dotted directly against the cached latent; $W_{i}^{U V}$ symmetrically absorbs into $W^{O}$ . You cache only $c_{s}^{K V}$ yet compute exact multi-head attention.

Decoupled RoPE — mandatory. Applying RoPE inserts a position rotation $R_{s - t}$ between $W_{i}^{U Q}$ and $W_{i}^{U K}$ ; it does not commute, so the absorbed matrix would depend on $(t, s)$ — killing absorption. Fix: a separate small key/query part carrying RoPE, shared MQA-style, added as a second score term:

$q_{t}^{(i) ⊤} k_{s}^{(i)} = absorbable, no position q_{t}^{C, (i) ⊤} k_{s}^{C, (i)} + RoPE, width d_{h}^{R} q_{t}^{R, (i) ⊤} k_{s}^{R} .$

Cache per token per layer: $d_{c} + d_{h}^{R}$ elements. $M_{KV}^{MLA} = L (d_{c} + d_{h}^{R}) s b p .$

Numbers. DeepSeek-V2 uses $d_{h} = 128$ , $d_{c} = 512$ , $d_{h}^{R} = 64$ → caches 576 elems/token/layer vs. MHA’s $2 n_{h} d_{h} = 32768$ ( $n_{h} = 128$ ): ~57× smaller, equal to GQA with only $g = 2.25$ groups, at MHA-level quality. DeepSeek-V3 (arXiv:2412.19437) carries it forward at scale [verify exact dims].

Why it matters

MLA is a strict improvement on GQA’s frontier — smaller cache than GQA-2.25 while retaining full multi-head expressivity — because the reduction comes from genuine low-rank K/V structure, not from discarding heads. It is the frontier KV-compression trick and a key enabler of DeepSeek’s economical long-context serving.

Taught in

05-mqa-gqa-mla — §3.3 down/up-projection, weight absorption, decoupled RoPE.

Gunjan Dhanuka — Learning Notes

Explorer

MLA (Multi-head Latent Attention)

MLA (Multi-head Latent Attention)

Definition

Key math / How it works

Why it matters

Taught in

See also

Graph View

Table of Contents

Backlinks