Perceiver Resampler / Q-Former

Definition

A Perceiver resampler (and its cousin the Q-Former) is a module in which a fixed bank of learned latent queries cross-attends to a variable-length input, emitting a fixed number of output vectors regardless of input size. It is the canonical way to compress a large, variable set of features (image patches, video frames) into a small, constant token budget before a downstream model. The lineage: Perceiver (arXiv:2103.03206) → Flamingo’s Perceiver resampler (arXiv:2204.14198) → Q-Former in BLIP-2 (arXiv:2301.12597).

Key math / How it works

Let $L\in {\mathbb{R}}^{r\times d}$ be a bank of $r$ learned latent queries and $Y\in {\mathbb{R}}^{m\times d}$ the variable-length input features:

$L^{\prime }=\text{XAttn}(\underset{r\times d\text{learned}}{\underbrace{L}},\underset{m\times d\text{input}}{\underbrace{Y}})\in {\mathbb{R}}^{r\times d},r\ll m.$

Because the queries are a fixed learned set and only the keys/values scale with the input, the output is always $r$ tokens. This:

• turns an $O(m^2)$ self-attention problem into $O(rm)$ cross-attention, and
• decouples downstream sequence length from input size — a video with many frames still emits $r$ tokens.

Concrete token budgets:

• Perceiver — a small set of latents cross-attend to a large byte/pixel array, decoupling depth/compute from input size.
• Flamingo resampler — compresses variable-length visual features to 64 latents.
• Q-Former (BLIP-2) — 32 learned query embeddings cross-attend to frozen image features, trained with image–text contrastive/matching/generation objectives so the 32 output vectors carry text-relevant content, then projected into the LLM input space → 32 tokens per image, independent of resolution.

Resampler ≈ Q-Former: the same architectural move — a fixed number of learned queries cross-attend to a variable number of features, emitting a fixed number of tokens.

Why it matters

In a VLM the visual token count drives the LM’s $O(s^2)$ attention (and linear KV-cache) cost. A resampler caps that count independent of resolution or frame count, so it is the highest-leverage lever for many-image and video regimes — and the reason native-resolution encoders (moonvit, navit-packing) can afford to emit thousands of patches before compression.

Taught in

• 08-attention-in-vits-and-vlms — §3.2 (Flamingo resampler), §3.3 (Q-Former/BLIP-2).

See also

• cross-attention — the underlying operation
• vlm-fusion — where the resampler sits in the fusion taxonomy
• vision-transformer — the encoder producing the features being compressed
• navit-packing · moonvit — native-resolution token explosion the resampler tames
• attention-mechanisms