Attention Glossary

Canonical vocabulary for the attention-mechanisms curriculum. Every lesson adheres to these definitions and this notation. Notation (fixed throughout): $s$ = sequence length, $d$ = model dim, $h$ = number of heads, $d_h=d/h$ = per-head dim, $b$ = batch size, $L$ = number of layers.

Core mechanism

• Query / Key / Value ($Q,K,V$) — The three learned projections of the input $X\in {\mathbb{R}}^{s\times d}$: $Q=X{W}_Q$, $K=X{W}_K$, $V=X{W}_V$. Queries probe, keys are matched against, values are aggregated. See 01-scaled-dot-product-attention.
• Attention score — The unnormalized query-key affinity $s_{ij}=q_i\cdot k_j/\sqrt{d_h}$; the matrix $S=Q{K}^{\top }/\sqrt{d_h}\in {\mathbb{R}}^{s\times s}$ before softmax. See 01-scaled-dot-product-attention.
• Scaled dot-product attention — The core operation $\mathrm{Attn}(Q,K,V)=\mathrm{softmax}\left(Q{K}^{\top }/\sqrt{d_h}\right)V$; a softmax-weighted average of values with weights set by query-key similarity. See 01-scaled-dot-product-attention.
• Softmax temperature / scaling — The $1/\sqrt{d_h}$ factor (equivalently temperature $\tau =\sqrt{d_h}$) that keeps score variance $O(1)$ so softmax gradients don’t vanish; lowering $\tau$ sharpens the distribution. See 01-scaled-dot-product-attention.
• Causal mask — An additive mask $M_{ij}=-\infty$ for $j>i$ applied to $S$ so position $i$ attends only to $j\le i$; enforces autoregressive factorization. See 01-scaled-dot-product-attention.
• Self-attention — Attention where $Q,K,V$ all derive from the same sequence, mixing information within one stream. See 01-scaled-dot-product-attention.
• Cross-attention — Attention where $Q$ comes from one stream and $K,V$ from another (e.g. decoder queries over encoder states); the mechanism behind encoder-decoder and multimodal conditioning. See 08-attention-in-vits-and-vlms.

Multi-head & blocks

• Multi-head attention (MHA) — Run $h$ attention operations in parallel on $d_h$-dim subspaces, then concatenate: $\mathrm{MHA}(X)=\mathrm{Concat}({\mathrm{head}}_1,\dots ,{\mathrm{head}}_h)W_O$. Lets different heads specialize. See 02-multi-head-attention-and-the-block.
• Head dimension ($d_h$) — Per-head width $d_h=d/h$; sets each head’s subspace size and the $\sqrt{d_h}$ scaling. See 02-multi-head-attention-and-the-block.
• Output projection ($W_O$) — The $d\times d$ matrix mixing concatenated head outputs back into the residual stream. See 02-multi-head-attention-and-the-block.
• Transformer block — One layer: attention sublayer + position-wise MLP, each wrapped in residual connection and LayerNorm; $L$ such blocks stacked. See 02-multi-head-attention-and-the-block.
• Pre-LN vs post-LN — Placement of LayerNorm relative to the residual add. Pre-LN ($x+f(\mathrm{LN}(x))$) gives clean residual gradient flow and stable deep training; post-LN ($\mathrm{LN}(x+f(x))$) is the original Transformer, better-performing but harder to train without warmup. See 02-multi-head-attention-and-the-block.

Positional

• Absolute vs relative position — Absolute encodings tag each token with its index $i$; relative encodings condition only on the offset $i-j$, generalizing better across lengths. See positional-encoding.
• Sinusoidal encoding — The original fixed absolute scheme adding ${\mathrm{PE}}_{i,2k}=\sin (i/10000^{2k/d})$ (and $\cos$) to embeddings; no learned parameters. See positional-encoding.
• RoPE (rotary position embedding) — Encodes position by rotating $q,k$ pairs by angle proportional to index, so $q_i\cdot k_j$ depends only on $i-j$; the de facto standard in modern LLMs. See rope.
• ALiBi — Adds a linear distance penalty $-m(i-j)$ to scores (per-head slope $m$) instead of positional embeddings; cheap length extrapolation. See positional-encoding.

Efficiency & KV-cache

• KV cache — Stored $K,V$ from all past positions during autoregressive decoding so each new token is $O(s)$ not $O(s^2)$; memory $\propto 2bsLh{d}_h$ and often the decode bottleneck. See kv-cache.
• Prefill vs decode — Prefill processes the whole prompt in parallel (compute-bound, high arithmetic intensity); decode generates one token at a time (memory-bound, dominated by KV-cache reads). See kv-cache.
• MQA (multi-query attention) — All $h$ query heads share a single $K,V$ head, shrinking the KV cache $h\times$ at some quality cost. See gqa-mqa.
• GQA (grouped-query attention) — Interpolates MHA and MQA: $g$ KV heads shared across groups of query heads ($1<g<h$); the common production choice. See gqa-mqa.
• MLA (multi-head latent attention) — Compresses $K,V$ into a low-rank latent cached per token, then up-projects per head at use time; cuts KV memory while retaining MHA-like quality (DeepSeek-V2/V3). See mla.
• Arithmetic intensity — FLOPs per byte of memory traffic; determines whether an attention kernel is compute- or memory-bound on the roofline. See arithmetic-intensity.

Sparse & linear

• Sliding-window attention — Each query attends only to the last $w$ keys, giving $O(sw)$ cost and a bounded KV cache; stacking layers grows the effective receptive field (e.g. Mistral, Longformer). See sparse-attention.
• Attention sink — Initial tokens that absorb disproportionate attention mass; retaining a few sink tokens alongside a sliding window (StreamingLLM) preserves quality under KV eviction. See sparse-attention.
• Sparse attention — Any pattern restricting each query to a subset of keys (strided, block, global+local) to break the $O(s^2)$ cost while approximating full attention. See sparse-attention.
• Linear attention — Replaces softmax with a kernel feature map $\varphi$ so $\mathrm{Attn}\approx \varphi (Q)(\varphi (K{)}^{\top }V)$, giving $O(s)$ cost and a recurrent state; trades exactness for scaling. See linear-attention.
• State-space model (SSM) — Sequence model with a linear recurrence $h_t=\bar{A}{h}_{t-1}+\bar{B}{x}_t$, $y_t=C{h}_t$; selective variants (Mamba) rival attention at $O(s)$ with constant-size state. See state-space-models.

Multimodal

• Perceiver / resampler — A cross-attention module where a small set of learned latent queries attends over a large (possibly variable-size) input, compressing it to a fixed token budget (Perceiver, Flamingo Perceiver-Resampler). See 08-attention-in-vits-and-vlms.

Long-context

• Context length — The maximum sequence $s$ a model attends over in one forward pass; the primary axis of the long-context frontier. See long-context.
• RoPE scaling / YaRN — Post-hoc rescaling of RoPE frequencies (position interpolation, NTK-aware, YaRN) to extend context beyond the trained length with minimal fine-tuning. See rope.
• Ring attention — Distributes the sequence across devices, passing $K,V$ blocks around a ring while overlapping compute and communication, enabling near-arbitrary context via online softmax accumulation. See long-context.

Systems

• FlashAttention — IO-aware exact attention kernel that tiles $Q,K,V$ into SRAM and uses online softmax to avoid materializing the $s\times s$ score matrix, cutting HBM traffic from $O(s^2)$ to $O(s)$. See flash-attention.
• Online softmax — Streaming softmax recurrence tracking running max $m$ and denominator $\ell$ so the normalization is computed in one pass without holding the full score row; the numerical core of FlashAttention and ring attention. See online-softmax.

---

Part of attention-mechanisms.