Cross-Attention

Definition

Cross-attention is scaled-dot-product attention in which the queries come from one sequence and the keys/values come from another. It is not a new operation — it is the same softmax-dot-product op as self-attention, which is simply the special case where $Q,K,V$ all come from the same sequence. Cross-attention is the mechanism by which one stream “reaches into” another: text into vision (VLM fusion), decoder into encoder (encoder–decoder Transformers), or learned latents into a variable-length input (Perceiver resampler).

Key math / How it works

Given a query stream $H\in {\mathbb{R}}^{s\times d}$ and a context stream $Y\in {\mathbb{R}}^{m\times d}$:

$\text{XAttn}(H,Y)=\mathrm{softmax}\left(\frac{(H{W}^Q)(Y{W}^K{)}^{\top }}{\sqrt{d_k}}\right)(Y{W}^V)\in {\mathbb{R}}^{s\times d}.$

• Output length tracks the query stream ($s$), not the context ($m$).
• Cost is $O(s\cdot m\cdot d)$ — asymmetric, unlike the $O(s^{2}d)$ of self-attention.
• Self-attention is the special case $H=Y$ (so $s=m$), giving the familiar $O(s^{2}d)$.

Gated cross-attention (Flamingo). When inserting cross-attention into a frozen LM, the output is added to the residual stream through a $\tanh$ gate initialized at 0:

$H\leftarrow H+\tanh (\alpha )\cdot \text{XAttn}(H,Y),\alpha \text{ init }0.$

At init $\tanh (0)=0$, so the augmented model is identical to the original LM — training starts from a known-good point and eases the new modality in gradually.

Why it matters

Cross-attention is the canonical way to condition one modality/stream on another without merging them into a single self-attention sequence. In VLMs it keeps the LM’s sequence length fixed while injecting vision as a side input (cross-attention fusion, Flamingo). In encoder–decoder models it lets the decoder attend to the full encoded source. And “fixed learned queries cross-attend to compress a variable-length input” is the single idea behind the Perceiver resampler and Q-Former.

Taught in

• 08-attention-in-vits-and-vlms — §3.2 (gated cross-attn in Flamingo); the “not a new op” warning.

See also

• vlm-fusion — cross-attention fusion vs. project-and-concatenate
• perceiver-resampler — learned-query cross-attention as compression
• vision-transformer — the vision features that become K/V
• flash-attention · kv-cache — the underlying attention kernel/cache
• attention-mechanisms