VLM Fusion (Cross-Attention vs. Project-and-Concatenate)

Definition

VLM fusion is how image information gets into a language model. Two families dominate. (1) Cross-attention fusion (Flamingo, arXiv:2204.14198): leave the LM’s token stream alone and insert new cross-attention layers where text queries attend to vision keys/values. (2) Project-and-concatenate (LLaVA, arXiv:2304.08485): project vision patch features into the LM’s embedding space with a small network and prepend them as extra tokens, so ordinary self-attention runs over one joint text+image sequence. Every VLM design is a variation on these two moves.

Key math / How it works

Cross-attention fusion (Flamingo). Frozen LM + frozen vision encoder, bridged by trainable gated cross-attention layers; visual features are first squeezed to a fixed count by a Perceiver resampler (64 latents):

$H\leftarrow H+\tanh (\alpha )\cdot \text{XAttn}(H,Y),\alpha \text{ init }0.$

LM sequence length stays $s$; per-layer image cost is $O(s\cdot m)$ with $m$ capped by the resampler.

Project-and-concatenate (LLaVA). A projector (linear in LLaVA-1, 2-layer MLP in LLaVA-1.5) maps patch features $Y\in {\mathbb{R}}^{n\times d_v}$ into the LM dim, then prepends them:

$\hat{Y}=\text{MLP}(Y)\in {\mathbb{R}}^{n\times d},z=[{\hat{y}}_1,\dots ,{\hat{y}}_n,e_1,\dots ,e_t].$

The LM runs ordinary causal self-attention over $s=n+t$; the projector is the only new module.

The tradeoff:

	Cross-attention fusion (Flamingo)	Project + concatenate (LLaVA)
New params	Many (gated x-attn, billions)	Few (a projector MLP)
LM seq length	Unchanged (vision is side input)	Grows by $n$ visual tokens
Per-layer image cost	$O(s\cdot m)$, $m$ fixed by resampler	folded into $O(s^2)$ self-attn
Flexibility (many img / video)	strong — resampler caps tokens	token budget blows up fast
Implementation	modify LM internals	leave LM untouched

Neither is strictly cheaper. Cross-attention pays in parameters and architectural surgery but caps visual tokens; concatenation is cheap to build but every visual token enters the LM’s quadratic. With $n_{\text{vis}}\gg t$, the LM’s attention cost $O(s^2)\approx O(n_{\text{vis}}^2)$ is driven almost entirely by the image token budget — e.g. a 20-token question about a 2304-token image spends ~98% of attention FLOPs on the image.

Why it matters

Concatenation won the open ecosystem: it reuses the LM unchanged and rides every LM improvement for free. But high-resolution / many-image / video regimes push back toward resampling because $n_{\text{vis}}$ hits the LM’s $O(s^2)$ attention and linear KV cache. Choosing a fusion strategy is fundamentally a choice about where the visual-token cost lives.

Taught in

• 08-attention-in-vits-and-vlms — §3.2–3.4, the fusion tradeoff table, and the FLOPs argument.

See also

• cross-attention · perceiver-resampler — the mechanisms of Family 1
• vision-transformer — the encoder feeding both families
• navit-packing · moonvit — native resolution and token-count explosion
• flash-attention · kv-cache — where the concatenated tokens’ cost lands
• attention-mechanisms