GQA / MQA (Grouped & Multi-Query Attention)

Definition

Multi-Query (Shazeer, 2019; arXiv:1911.02150) and Grouped-Query Attention (Ainslie et al., 2023; arXiv:2305.13245) shrink the KV cache by decoupling the number of query heads from the number of distinct K/V heads $n_{kv}$: multiple query heads share the same keys and values. MQA collapses to $n_{kv}=1$; GQA uses $1<g<h$ groups, the production sweet spot (Llama-2/3-class, most modern LLMs).

Key math / How it works

The KV cache is $M_{\text{KV}}=2L{n}_{kv}{d}_{h}sbp$ bytes. Of $\{2,L,n_{kv},d_h,s,b,p\}$, $n_{kv}$ is the one architectural knob, so it is the lever:

• MHA: $n_{kv}=h$ — every query head has a private K/V head. Max cache.
• MQA: $n_{kv}=1$ — all query heads share one K/V head. Cache $÷h$, but collapsing $h$ subspaces into one costs quality and can be training-unstable at scale.
• GQA: partition $h$ query heads into $g$ groups ($g\mid h$), each with a shared K/V head. $M_{\text{KV}}\propto g$; $g=h$ recovers MHA, $g=1$ recovers MQA.

Two GQA findings make it ubiquitous: (1) the quality-vs-$g$ curve is steep near $g=1$ and flat by a handful of groups, so $g\approx 8$ recovers ~all MHA quality at most of MQA’s savings; (2) uptraining — convert an existing MHA checkpoint by mean-pooling each group’s K/V projections and fine-tuning for ~5% of pretraining compute, no from-scratch pretrain. Grouping is stable where MQA is not because each shared head serves only a handful of similar query heads.

FLOPs vs. bytes. This is a decode-memory optimization, not a FLOPs one: $Q{K}^{\top }$ and $A\cdot V$ still sum over full width $d$ (the shared K/V head broadcasts to all query heads), so prefill FLOPs are ~unchanged; only the small K/V projection GEMMs shrink. What drops is KV-cache capacity and the memory-bandwidth traffic of memory-bound decode. E.g. GQA-8 on a 64-head model: 8× smaller cache than MHA at ~flat prefill FLOPs.

Why it matters

GQA is the default KV reduction in nearly all production LLMs — near-MHA quality at $\sim h/g$ less cache, letting servers fit more concurrent sequences and longer context. It is the baseline that MLA later dominates by attacking the width of what is cached rather than the number of heads.

Taught in

• 05-mqa-gqa-mla — §3.1–3.2 MQA/GQA math, uptraining, the sweet spot.

See also

• mla — the low-rank successor · kv-cache — the constraint
• paged-attention · prefill-decode · quantization-inference — the orthogonal $p$ lever
• attention-glossary · attention-mechanisms