Vision Transformer (ViT)

Definition

The Vision Transformer (ViT) (Dosovitskiy et al., 2020; arXiv:2010.11929) applies a standard Transformer encoder directly to images by treating fixed-size image patches as the token sequence. No convolutions: chop the image into a grid, linearly embed each patch, add position embeddings, and run self-attention over the patches. The same scaled-dot-product attention op from language transfers unchanged — only the tokenizer (patchify + linear embed) is new.

Key math / How it works

Given an image $x\in {\mathbb{R}}^{H\times W\times C}$ and patch size $p$, you get $n_{\text{patch}}=HW/p^2$ patches. Flatten each to ${\mathbb{R}}^{p^{2}C}$ and linearly embed with $E\in {\mathbb{R}}^{(p^{2}C)\times d}$:

$z_0=[x_{\text{cls}};x_p^{1}E;\dots ;x_p^{n}E]+E_{\text{pos}}.$

• A learnable [CLS] token $x_{\text{cls}}$ is prepended; after $L$ standard blocks its final state is the image representation for classification.
• Position embeddings $E_{\text{pos}}$ are required because attention is permutation-invariant — it must be told where each patch sits. (Contrast the 2D-RoPE / interpolated-absolute treatment in MoonViT.)
• Each block is ordinary multi-head self-attention over the $n_{\text{patch}}+1$ tokens, then an MLP.

The attention core costs

$\approx 4b{n}_{\text{patch}}^{2}d\text{FLOPs, i.e. }O(n_{\text{patch}}^{2}d).$

Patch count is a resolution knob with quadratic teeth. Halving patch size $p$ quadruples $n_{\text{patch}}$ and 16בs the attention compute. Concretely, a $224\times 224$ image at $p=16$ gives $196$ patches; a $1920\times 1080$ image at $p=14$ gives $\approx 10,500$ patches before any compression.

Why it matters

In a language model, sequence length is set by the text. In a ViT, “sequence length” is set by resolution ÷ patch size — a design choice with quadratic cost. This is why native-/any-resolution encoders (navit-packing, moonvit) bolt on aggressive token compression, and why the visual token budget dominates the attention bill of a downstream VLM. ViT is the foundation of nearly every modern vision encoder and multimodal model.

Taught in

• 08-attention-in-vits-and-vlms — §3.1 patch embedding, [CLS], the quadratic-in-patch-count cost.
• moonvit — native-resolution successor with windowed 2D-RoPE attention.

See also

• cross-attention — how vision features enter a language model
• perceiver-resampler · vlm-fusion — compressing/fusing patch tokens
• navit-packing · moonvit · global-native-encoding — native-resolution encoding
• flash-attention — the kernel that makes high-patch-count attention affordable
• positional-encoding · rope — position information for patches
• attention-mechanisms