Learn: Attention in ViTs & VLMs

What you're learning

The exact same attention op from Lesson 01 — but pointed at image patches instead of word tokens, and then wired into a language model. By the end you’ll know the two dominant ways image information enters an LLM (cross-attention fusion vs. project-and-concatenate), the architectures that exemplify each (Flamingo, BLIP-2/Q-Former, LLaVA), and — the punchline — why the number of visual tokens is the quantity that dominates a modern VLM’s attention bill.

First lesson of Tier 3. Assumes Tiers 1–2: scaled dot-product attention (01-scaled-dot-product-attention), the block (02-multi-head-attention-and-the-block), the $O(s^2)$ wall (04-quadratic-bottleneck-and-io-aware), and the subquadratic escape routes (07-linear-and-subquadratic). Vocabulary in attention-glossary.

---

1. Learning map

graph TD
    A["SDPA (Lesson 01)<br/>attention over a token sequence"] --> B["ViT: treat image patches<br/>AS the token sequence"]
    B --> C["Multimodal question:<br/>how does vision get INTO an LLM?"]
    C --> D1["Family 1: CROSS-ATTENTION fusion<br/>LM tokens attend to vision features"]
    C --> D2["Family 2: PROJECT + CONCATENATE<br/>vision tokens joined into the stream"]
    D1 --> E1["Flamingo<br/>gated x-attn + Perceiver resampler"]
    D1 --> E2["BLIP-2 / Q-Former<br/>learned queries → fixed tokens"]
    D2 --> F1["LLaVA<br/>MLP projector, prepend as tokens"]
    F1 --> G["Native / any-resolution<br/>token-count explosion"]
    E1 --> G
    E2 --> G
    G --> H["FLOPs: visual tokens<br/>drive the O(s²) cost"]
    style B fill:#44a,color:#fff
    style D1 fill:#4a4,color:#fff
    style D2 fill:#a64,color:#fff
    style H fill:#a44,color:#fff

---

2. Intuition

An image is just a sequence of patches

Attention doesn’t know or care that its inputs are words. It takes a set of vectors and mixes them by learned query–key similarity (Lesson 01). So to run a Transformer on an image, you only need to turn the image into a set of vectors. The Vision Transformer (ViT) does the obvious thing: chop the image into a grid of fixed-size patches (say $16\times 16$ pixels), flatten and linearly embed each patch into a $d$-dimensional vector, and hand that sequence to a standard Transformer encoder.¹ Patches become “tokens,” self-attention lets every patch look at every other patch, and — that’s it. No convolutions required.

The one-sentence version

A ViT is a plain Transformer whose “tokens” are image patches. Everything you learned about attention transfers unchanged; only the tokenizer (patchify + linear embed) is new.

Now the multimodal question

Suppose you have a good vision encoder (a ViT that turns an image into $n$ patch feature vectors) and a good language model. How do you get the image information into the LM so it can answer questions about the picture? Two families dominate:

• Family 1 — Cross-attention fusion. Leave the LM’s token stream alone. Insert new cross-attention layers where the LM’s text tokens issue queries and the keys/values come from the vision features. Vision is a side input the text “reaches into.” (Flamingo.)²
• Family 2 — Project and concatenate. Project the vision patch features into the LM’s embedding space with a small network, then prepend them as if they were extra tokens. The LM’s ordinary self-attention now runs over one joint text+image sequence — no new layer type. (LLaVA.)³

Everything in this lesson is a variation on these two moves. The tension between them — parameters, flexibility, and cost — is the thing to hold in your head.

"Cross-attention" is not a new operation

It’s the same softmax-dot-product op. The only change is that $Q$ comes from one sequence (text) while $K,V$ come from another (vision). Self-attention is the special case $Q,K,V$ from the same sequence. Don’t reify it into something exotic.

---

3. Math / architectures

3.1 ViT — self-attention over patches

Given an image $x\in {\mathbb{R}}^{H\times W\times C}$ and patch size $p$, you get $n_{\text{patch}}=\frac{HW}{p^2}$ patches. Flatten each to ${\mathbb{R}}^{p^{2}C}$ and linearly embed:

$z_0=[x_{\text{cls}};x_p^{1}E;x_p^{2}E;\dots ;x_p^{n}E]+E_{\text{pos}},E\in {\mathbb{R}}^{(p^{2}C)\times d}$

A learnable [CLS] token $x_{\text{cls}}$ is prepended; after $L$ standard Transformer blocks its final state is used as the image representation for classification.¹ Position embeddings $E_{\text{pos}}$ are added because attention is permutation-invariant — it needs to be told where each patch sits (contrast the 2D-RoPE / interpolated-absolute treatment in MoonViT).

Each block is exactly Lesson 02: multi-head self-attention over the $n_{\text{patch}}+1$ tokens, then an MLP. The cost is the familiar quadratic:

$\text{attention core}\approx 4b{n}_{\text{patch}}^{2}d\text{FLOPs, i.e. }O(n_{\text{patch}}^{2}d).$

The headline: patch count is a resolution knob with quadratic teeth. Halving patch size $p$ quadruples $n_{\text{patch}}$ and 16בs the attention compute. This is the same $O(s^2)$ wall from 04-quadratic-bottleneck-and-io-aware, now indexed by image resolution.

Key insight

In an LM, sequence length is set by the text. In a ViT, “sequence length” is set by resolution ÷ patch size — a design choice you make. That choice is the entire reason native-resolution encoding (Section 3.5) is hard.

3.2 Cross-attention fusion — Flamingo

Flamingo keeps a frozen pretrained LM and a frozen vision encoder, and interleaves new trainable gated cross-attention layers between the LM’s existing blocks.² At such a layer, the text hidden states $H\in {\mathbb{R}}^{s\times d}$ produce queries; the vision features $Y\in {\mathbb{R}}^{m\times d}$ produce keys and values:

$\text{XAttn}(H,Y)=\mathrm{softmax}\left(\frac{(H{W}^Q)(Y{W}^K{)}^{\top }}{\sqrt{d_k}}\right)(Y{W}^V)\in {\mathbb{R}}^{s\times d}.$

The output is added back to the residual stream through a tanh gate initialized at $0$:

$H\leftarrow H+\tanh (\alpha )\cdot \text{XAttn}(H,Y),\alpha \text{ init }0.$

At init $\tanh (0)=0$, so the augmented model is identical to the original LM — training starts from a known-good point and eases vision in gradually.² This is the defining trick of gated fusion.

The Perceiver resampler. A raw image (or a video with many frames) yields a variable and often large number of vision features $m$. Flamingo first squeezes them to a fixed small set (64 latents) with a Perceiver-style module: a fixed bank of $r$ learned latent queries cross-attends to the variable-length vision features.²⁴⁵

$L^{\prime }=\text{XAttn}(\underset{r\times d\text{learned}}{\underbrace{L}},\underset{m\times d\text{visual}}{\underbrace{Y}})\in {\mathbb{R}}^{r\times d},r\ll m.$

Because the queries are a fixed learned set and the keys/values scale with input, the output length is pinned to $r$ regardless of how big or numerous the images are — turning a $O(m^2)$ problem into $O(rm)$ and decoupling downstream cost from input size. This “learned queries cross-attend to compress” pattern is the ancestor of the Q-Former.

3.3 Q-Former / BLIP-2

BLIP-2 bridges a frozen image encoder and a frozen LLM with a lightweight Querying Transformer (Q-Former).⁶ A small fixed set of learned query embeddings (32 in BLIP-2) cross-attend to the frozen image features to extract a fixed-length visual summary; those queries are then projected into the LLM’s input space. Same skeleton as the resampler — learned queries $\to$ cross-attention $\to$ fixed token count — trained with image–text contrastive/matching/generation objectives so the 32 output vectors carry text-relevant visual content. The result: 32 tokens per image into the LLM, independent of resolution.

Resampler ≈ Q-Former, one idea

Perceiver resampler and Q-Former are the same architectural move: a fixed number of learned queries cross-attend to a variable number of visual features, emitting a fixed number of visual tokens. Memorize this once; it recurs everywhere (token-compression).

3.4 Projection / concatenation — LLaVA

LLaVA throws away the extra machinery. Take a frozen CLIP ViT, grab its patch features $Y\in {\mathbb{R}}^{n\times d_v}$, pass them through a small projector — in LLaVA-1 a single linear layer, in LLaVA-1.5 a 2-layer MLP — to land in the LM embedding dimension, then prepend the projected features as tokens in front of the text embeddings:³⁷

$\hat{Y}=\text{MLP}(Y)\in {\mathbb{R}}^{n\times d},z=[{\hat{y}}_1,\dots ,{\hat{y}}_n,e_1,\dots ,e_t].$

Now the LM runs ordinary causal self-attention over the joint length $s=n+t$. No new layer type, no gates, no resampler — the projector is the only new module (a few million params vs. Flamingo’s billions of cross-attention params).

Why concatenation won the open ecosystem

It reuses the LM unchanged, is trivial to implement, and lets vision tokens interact with text through the same attention the LM already trained — so it rides every LM improvement for free. The cost, as Section 5 shows, is that every visual token now lives inside the LM’s $O(s^2)$ attention.

Cross-attention vs. concatenate — the tradeoff table:

	Cross-attention fusion (Flamingo)	Project + concatenate (LLaVA)
New params	Many (gated x-attn blocks, billions)	Few (a projector MLP)
LM sequence length	Unchanged (vision is side input)	Grows by $n$ visual tokens
Per-layer image cost	$O(s\cdot m)$ cross-attn, $m$ fixed by resampler	folded into $O(s^2)$ self-attn
Where cost lives	extra layers, but $m$ is capped	the LM’s own attention, scales with $n$
Flexibility (many images / video)	strong — resampler caps tokens per image	token budget blows up fast
Implementation	modify LM internals	leave LM untouched

Neither is strictly "cheaper"

Cross-attention pays in parameters and architectural surgery but caps visual tokens via the resampler. Concatenation pays in sequence length — cheap to build, but $n$ visual tokens enter the LM’s quadratic. Which wins depends on $n$, number of images, and whether you can modify the LM. High-resolution / many-image / video regimes push you back toward resampling.

3.5 Native / any-resolution — where token count explodes

Fixed $224\times 224$ ViTs waste a real image’s detail. Native-resolution encoders (NaViT packing, MoonViT) instead encode the image at its true size in one global pass — see the full treatment in moonvit and the concept page moonvit, with packing in navit-packing and the design axis in global-native-encoding. (Don’t re-derive it here.)

The consequence that matters for this lesson: native resolution means $n_{\text{patch}}$ scales with actual image area, so a single high-res image can be thousands of patch tokens. A $1920\times 1080$ image at patch 14 is $\approx 10,500$ tokens before compression.⁸ In a concatenate-style VLM those tokens land directly in the LM sequence. That is precisely why native-resolution encoders bolt on aggressive token compression — pixel-shuffle, resamplers, Q-Formers — before the LM (covered in token-compression). The vision side’s resolution ambition is what makes the cross-modal cost problem acute.

---

4. Illustration — the two fusion families

flowchart TB
    IMG["Image"] --> VE["Vision encoder (ViT)<br/>→ n patch features"]

    subgraph XA["Family 1: Cross-attention fusion (Flamingo / BLIP-2)"]
        direction TB
        VE --> R["Resampler / Q-Former<br/>learned queries cross-attend<br/>→ fixed m tokens (e.g. 32-64)"]
        R --> XL["LM block"]
        TXT1["Text tokens (length s)"] --> XL
        XL -.->|"gated cross-attn:<br/>Q=text, K,V=vision"| XL
        XL --> OUT1["LM output"]
    end

    subgraph PC["Family 2: Project + concatenate (LLaVA)"]
        direction TB
        VE --> P["Projector MLP<br/>→ n tokens in LM dim"]
        P --> CAT["Concatenate: [vision(n) ; text(t)]"]
        TXT2["Text tokens (length t)"] --> CAT
        CAT --> SL["LM blocks<br/>self-attn over s = n + t"]
        SL --> OUT2["LM output"]
    end

    style XA fill:#153,color:#fff
    style PC fill:#420,color:#fff

Read the contrast: in Family 1 the LM sequence stays length $s$ and vision enters through dedicated cross-attention fed by a length-capped resampler. In Family 2 vision becomes $n$ extra tokens and rides the LM’s existing self-attention over the joint length $n+t$.

---

5. FLOPs — visual tokens are the cost driver

ViT side. Self-attention core is $O(n_{\text{patch}}^{2}d)$, quadratic in patch count (Section 3.1). Compression before the LM (pixel-shuffle, resampler) is a deliberate attack on this term.

VLM side — the important part. In a concatenate-style decoder VLM, the LM’s per-layer attention is $O(s^{2}d)$ with $s=n+t$. Text prompts are often short (tens–hundreds of tokens); a single high-res image can be hundreds to thousands of visual tokens. So $n$ dominates $s$, and the LM’s attention cost is driven almost entirely by the image token budget:

$s=n_{\text{vis}}+t,n_{\text{vis}}\gg t\Rightarrow O(s^2)\approx O(n_{\text{vis}}^2).$

Multiple images or video frames multiply $n_{\text{vis}}$ linearly, and it hits the LM’s quadratic. This is the whole economic argument for resamplers, Q-Formers, and pixel-shuffle: they shrink $n_{\text{vis}}$, and every token removed is paid back quadratically inside the LM (and linearly in the KV cache, 05-mqa-gqa-mla).

Reframe the calculator

The Attention FLOPs & KV-cache calculator is “just” an $O(s^2)$ attention estimator — but for a VLM, read $s$ as visual tokens + text tokens. Set $s$ to your image-token budget (e.g. 576 for LLaVA-1.5’s $24^2$ CLIP grid, or 2000+ for a native-res image) and the prefill FLOPs it reports are your image’s attention cost inside the LM.

Quiz (retrieval): Cross-attention fusion vs. project-and-concatenate — name two axes on which they trade off, and say which way each leans.

Answer

(1) Parameters / architectural change: cross-attention adds many new params and surgically modifies the LM; concatenation adds only a tiny projector and leaves the LM untouched. (2) Sequence length / where cost lives: concatenation grows the LM sequence by $n$ visual tokens, so image cost lives inside the LM’s $O(s^2)$ self-attention; cross-attention keeps the LM sequence fixed and caps visual tokens via a resampler, paying instead in extra cross-attn layers. (Also acceptable: flexibility for many-image/video — resampling scales more gracefully.)

Quiz (retrieval): In a LLaVA-style VLM answering a 20-token question about one high-res image encoded to 2304 visual tokens, roughly what fraction of the LM's per-layer attention FLOPs is "about the image," and why?

Answer

Essentially all of it. $s=2304+20=2324$; attention $\propto s^2$. The image alone (ignoring cross terms) accounts for $\sim (2304/2324{)}^2\approx 98\%$ of the quadratic. Because attention cost grows with the square of sequence length and visual tokens vastly outnumber text tokens, the image dominates — which is exactly why shrinking $n_{\text{vis}}$ (pixel-shuffle, resampler, Q-Former) is the highest-leverage optimization in decoder VLMs.

Quiz (retrieval): Why does Flamingo's Perceiver resampler use learned queries whose count is fixed, rather than using the image features as queries?

Answer

The number of image features $m$ is variable (aspect ratio, resolution, number of frames) and often large. If image features were the queries, output length would track $m$ and cost would stay $O(m^2)$. With a fixed bank of $r$ learned queries cross-attending to the $m$ features, the output is always $r$ tokens ($r\ll m$) at $O(rm)$ cost — decoupling downstream sequence length from input size. Q-Former uses the identical trick with 32 queries.

---

6. Practice

Problem: added attention cost of $N$ image tokens

A decoder VLM has a 20-token text prompt. You feed it one image. Compare the LM’s per-layer attention-core FLOPs (use $\approx 4b{s}^{2}d$, $b=1$, $d=4096$) for three encodings: (a) Q-Former, 32 tokens; (b) LLaVA-1.5 grid, 576 tokens; (c) native-res, 2304 tokens. What’s the ratio (c)/(a)?

Solution. $s_a=52,s_b=596,s_c=2324$.

• (a) $4\cdot 52^2\cdot 4096\approx 4.43\times 10^7$ FLOPs
• (b) $4\cdot 596^2\cdot 4096\approx 5.82\times 10^9$
• (c) $4\cdot 2324^2\cdot 4096\approx 8.85\times 10^{10}$

Ratio (c)/(a) $\approx 2000\times$. The image token budget, not the model width or the text, sets the attention bill. Same op, same $d$ — only $s$ changed. (This is per layer; multiply by depth and batch. And the KV cache grows linearly in these same token counts.)

Problem: does compression pay for itself?

Native encoding gives 2304 visual tokens; a 2×2 pixel-shuffle cuts it 4× to 576. The shuffle+projector is negligible FLOPs. By what factor does the LM’s per-layer attention drop (text ≈ 0)?

Solution. Attention $\propto n_{\text{vis}}^2$, so $4\times$ fewer tokens $\Rightarrow 4^2=16\times$ less attention compute (and $4\times$ less KV-cache memory). A cheap 4× token cut buys a 16× attention saving inside the LM — the quadratic is why compression modules exist. See moonvit §5 for the pixel-shuffle mechanics.

---

7. Read next / primary sources

Primary sources, in reading order:

1. ViT — Dosovitskiy et al., “An Image Is Worth 16×16 Words,” arXiv:2010.11929.¹ Read §3.1 (patch embedding, [CLS], the “standard Transformer” claim).
2. LLaVA — Liu et al., arXiv:2304.08485;³ and LLaVA-1.5, arXiv:2310.03744⁷ for the MLP projector and the 576-token grid.
3. Flamingo — Alayrac et al., arXiv:2204.14198² — gated cross-attention + Perceiver resampler; the Perceiver lineage is arXiv:2103.03206⁴ / Perceiver IO arXiv:2107.14795.⁵
4. BLIP-2 / Q-Former — Li et al., arXiv:2301.12597.⁶
5. Native resolution — cross-links: moonvit, navit-packing, global-native-encoding.

Ask your teacher

Want the gated-cross-attention gradient flow worked out, a video-VLM token-budget analysis, or a side-by-side of Qwen-VL / InternVL / Kimi-VL fusion choices? Ask — I can go deeper on any piece or hand you a harder problem. [verify] any current-generation VLM specifics against the latest tech reports; production designs shift fast.

What’s next

• 09-long-context-race — visual tokens are one big driver of long sequences; the next lesson is the general fight for context length.
• Back to 07-linear-and-subquadratic — the subquadratic methods that would tame $O(n_{\text{vis}}^2)$ if quality held up.
• Sideways: token-compression (stub) — pixel-shuffle, ToMe, resamplers, PVC as the frontier of cheap visual tokens.

---

References

Reference: attention-glossary | Topic: attention-mechanisms | Filed: 2026-09-10

Footnotes

1. Dosovitskiy et al., “An Image Is Worth 16×16 Words: Transformers for Image Recognition at Scale,” arXiv:2010.11929 (2020). https://arxiv.org/abs/2010.11929 — patchify + linear embed + learnable [CLS] + position embeddings, then a standard Transformer encoder over patch tokens. [established] ↩ ↩² ↩³

2. Alayrac et al., “Flamingo: a Visual Language Model for Few-Shot Learning,” arXiv:2204.14198 (2022). https://arxiv.org/abs/2204.14198 — frozen LM + frozen vision encoder bridged by trainable gated cross-attention layers ($\tanh$-gate init 0); Perceiver resampler compresses variable-length visual features to 64 latents via cross-attention. [established] ↩ ↩² ↩³ ↩⁴ ↩⁵

3. Liu et al., “Visual Instruction Tuning” (LLaVA), arXiv:2304.08485 (2023). https://arxiv.org/abs/2304.08485 — a projection (linear in v1) maps frozen CLIP ViT patch features into the LM embedding space; visual tokens are prepended to text and attended by ordinary self-attention. [established] ↩ ↩² ↩³

4. Jaegle et al., “Perceiver: General Perception with Iterative Attention,” arXiv:2103.03206 (2021). https://arxiv.org/abs/2103.03206 — a fixed set of learned latents cross-attend to a large input array, decoupling depth/compute from input size. [established] ↩ ↩²

5. Jaegle et al., “Perceiver IO: A General Architecture for Structured Inputs & Outputs,” arXiv:2107.14795 (2021). https://arxiv.org/abs/2107.14795 — the resampler/decoder generalization Flamingo’s design draws on. [established] ↩ ↩²

6. Li et al., “BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models,” arXiv:2301.12597 (2023). https://arxiv.org/abs/2301.12597 — the Querying Transformer (Q-Former) uses 32 learned query embeddings that cross-attend to frozen image features to emit a fixed-length visual token set for a frozen LLM. [established] ↩ ↩²

7. Liu et al., “Improved Baselines with Visual Instruction Tuning” (LLaVA-1.5), arXiv:2310.03744 (2023). https://arxiv.org/abs/2310.03744 — upgrades the projector to a 2-layer MLP; the CLIP-ViT-L/14-336 grid yields $24\times 24=576$ visual tokens per image. [established] ↩ ↩²

8. MoonViT-SO-400M model card / config.json, Moonshot AI. https://huggingface.co/moonshotai/MoonViT-SO-400M — patch size 14, hidden 1152; a $1920\times 1080$ image $\Rightarrow \lceil 1920/14\rceil \times \lceil 1080/14\rceil \approx 10,549$ patch tokens before pixel-shuffle compression. See moonvit. [recent] ↩