Learn: MoonViT

What you're learning

How MoonViT encodes arbitrary-resolution images in a single global transformer pass — and why that’s non-trivial. By the end you should be able to derive the key design choices from first principles.

---

1. Learning map

The path to understanding MoonViT bottom-up:

graph TD
    A["Standard ViT<br/>(fixed resolution)"] --> B["High-res problem<br/>in MLLMs"]
    B --> C1["Option A: Resize<br/>→ loses detail"]
    B --> C2["Option B: Slice-Based Encoding<br/>→ breaks cross-region context"]
    B --> C3["Option C: Global Native Encoding<br/>← MoonViT's bet"]
    C3 --> D["NaViT packing<br/>(variable-length batching)"]
    C3 --> E["Dual positional embeddings<br/>(absolute + 2D RoPE)"]
    D --> F["FlashAttention varlen kernel"]
    E --> G["Resolution generalization"]
    F --> H["MoonViT"]
    G --> H
    H --> I["MLP Projector<br/>(pixel shuffle → LLM)"]

    style H fill:#4a4,color:#fff
    style C3 fill:#44a,color:#fff

---

2. Why native resolution matters

Common framing mistake

It’s tempting to think high-res is about “more pixels = more information.” That’s not the core issue. The issue is spatial coherence: tasks like OCR, chart reading, and dense visual QA require the model to reason about relationships across different parts of the image simultaneously. Slicing destroys that.

Standard ViTs (SigLIP, CLIP, etc.) are trained at fixed resolution — typically 224×224 or 384×384.¹ Two problems arise when you scale to real images in an MLLM:

Problem 1: Resize loses structure.
A 1920×1080 → 384×384 resize throws away $\frac{1920\times 1080}{384\times 384}\approx 14\times$ the pixel content. Small text, fine-grained charts, and dense diagrams become unreadable.

Problem 2: Slice-Based Encoding (SBE) breaks global context.
SBE (used by LLaVA, InternVL, etc.) tiles the image into $N$ fixed crops, encodes each independently, concatenates tokens.² A word that spans two tiles, or a caption spatially separated from its figure, gets cut apart — the encoder never sees their relationship.

Quiz: Why can't you just stitch SBE crop tokens back together in the LLM?

Answer

The LLM does see all the tokens, but the vision encoder has already processed each crop in isolation. Cross-region features (e.g. attention patterns that span the crop boundary) are never computed. By the time the LLM sees the tokens, the local visual representations are already “blind” to their neighbors in adjacent crops. The LLM would need to re-derive visual relationships purely from token co-occurrence — something it’s not specialized for.

---

3. The solution: sequence packing (NaViT)

The core enabling idea: concatenate, don’t pad.³ MoonViT adopts this NaViT packing scheme directly.⁴

Standard batching (broken for variable resolution)

To batch images of different sizes in a standard ViT, you pad shorter sequences to the longest:

Image A (196 tokens): [t1 t2 ... t196] [PAD PAD ... PAD]  ← wasted compute
Image B (9800 tokens): [t1 t2 ... t9800]

All padding tokens run through the full transformer and contribute nothing. At high resolution the waste is severe.

NaViT packing

Instead, concatenate all images into one flat sequence and track boundaries:

Packed: [img_A_t1 ... img_A_t196 | img_B_t1 ... img_B_t9800]
cu_seqlens: [0, 196, 9996]

cu_seqlens is a cumulative-sum tensor of sequence lengths. FlashAttention’s varlen kernel uses it to enforce block-diagonal attention⁵ — tokens from image A attend only to image A, tokens from image B attend only to image B:

$A_{ij}=\{\begin{array}{ll}\text{softmax}\left(\frac{q_i^{\top }{k}_j}{\sqrt{d}}\right)& \text{if }i,j\text{ in same image}\\ -\infty & \text{otherwise}\end{array}$

No explicit masking matrix is materialized. FlashAttention handles this in the CUDA kernel itself — efficient at scale.

Quiz: What's the memory complexity of NaViT packing vs. padded batching, for a batch of images with total tokens $N_{total}$ and max tokens $N_{max}$?

Answer

• Padded: $O(B\times N_{max}^2)$ attention — you pay for the largest image times batch size
• Packed NaViT: $O({\sum }_i{N}_i^2)$ — you pay for each image’s own quadratic cost, no inter-image waste
  In the typical case where images have very different sizes (a mix of small crops and large images), packing is significantly cheaper. In the worst case (all images same size), they’re equivalent.

---

4. Positional embeddings: why you need both

This is the most subtle design choice in MoonViT. Neither approach alone is sufficient.

Absolute positional embeddings (from SigLIP init)

SigLIP was trained with learned absolute positional embeddings — a lookup table of shape $(H_{train}/p,W_{train}/p,d)$ where $p=14$ is the patch size.¹ At inference on a different resolution, these embeddings are bicubically interpolated to the new grid⁴:

${\text{pos\_embed}}_{new}(h,w)=\text{bicubic\_interp}\left({\text{pos\_embed}}_{train},\frac{h\cdot H_{train}}{H_{new}},\frac{w\cdot W_{train}}{W_{new}}\right)$

This preserves the pretrained SigLIP signal — crucial for a smooth initialization. But it degrades for large resolution changes because bicubic interpolation assumes the learned embeddings vary smoothly, which they don’t always do.

2D Rotary Position Embeddings (RoPE)

RoPE encodes position in the attention computation itself, not as an additive token feature.⁶ For 1D RoPE:

$\text{Attn}(q_m,k_n)=(R_m{q}_m{)}^{\top }(R_n{k}_n)=q_m^{\top }{R}_m^{\top }{R}_n{k}_n=q_m^{\top }{R}_{m-n}{k}_n$

where $R_{\theta }$ is a block-diagonal rotation matrix. The key property: the dot product depends only on the relative position $m-n$, not absolute positions. This generalizes to unseen sequence lengths.

For 2D (height × width), MoonViT factorizes: half the head dimensions carry height-RoPE, the other half carry width-RoPE⁴:

$R_{(h,w)}=R_h\otimes R_w$

Why not just RoPE alone?

Starting from SigLIP with only RoPE would require extensive retraining to unlearn the absolute-embedding signal baked into the weights. Keeping both lets you use SigLIP as-is, then layer RoPE on top to handle resolution generalization. The Kimi-VL paper shows both together outperform either alone.⁴

Quiz: At a high resolution never seen during SigLIP training (e.g. 2048×2048), which positional embedding carries more weight — the interpolated absolute embeds or 2D RoPE? Why?

Answer

2D RoPE carries more weight at extreme out-of-distribution resolutions. Bicubic interpolation of absolute embeds degrades as the source-to-target grid ratio increases — the spatial structure becomes smoothed out and loses precision. RoPE, by contrast, encodes relative positions through a parameterization that generalizes by construction (the rotation matrix $R_{m-n}$ is computed analytically from the position, not looked up). So: at moderate resolution changes, both contribute; at large changes, RoPE dominates.

---

5. Architecture walk-through

graph LR
    subgraph Input
        A["Image H×W×3"]
    end
    subgraph Encoder
        B["PatchEmbed\nConv2d(3→1152, k=14, s=14)\n→ (H/14·W/14) tokens"]
        C["AbsPosEmbed\nbicubic interp to grid"]
        D["×27 EncoderLayers\nLayerNorm → Attn+2DRoPE → MLP"]
        E["LayerNorm\nout: (N_tok, B, 1152)"]
    end
    subgraph Projector
        F["Pixel Shuffle 2×2\n4× token reduction\n1152 → 4608"]
        G["Linear → GELU → Linear\n4608 → d_llm"]
    end
    A --> B --> C --> D --> E --> F --> G

Key numbers:

• Patch size: 14px → each patch is a 14×14 grid cell⁷
• A 1920×1080 image: $\lceil 1920/14\rceil \times \lceil 1080/14\rceil =137\times 77=10549$ tokens before pixel shuffle, $\approx 2637$ after
• Hidden dim: 1152
• 27 transformer layers
• MLP: a standard transformer FFN. The SigLIP-SO-400M base uses a GELU-tanh activation (gelu_pytorch_tanh), and MoonViT’s public config does not override it⁸

Quiz: Why pixel shuffle (space-to-depth) rather than average pooling for token compression?

Answer

Pixel shuffle (2×2 space-to-depth) merges each 2×2 block of spatial tokens into one token by concatenating their feature vectors: $(h,w,d)\to (h/2,w/2,4d)$. This is lossless — all 4 tokens’ information is preserved in the merged token, just at higher dimension. Average pooling would throw away spatial detail by averaging. The merged token is then projected down to $d_{llm}$ by the MLP. For an MLLM where the LLM needs to reason about fine-grained visual details, lossless merging is preferable.

---

6. Training

timeline
    title MoonViT Training Phases
    Phase 1 (2T tokens) : CoCa-style
                        : SigLIP contrastive loss
                        : Caption generation loss
                        : MoonViT + text decoder updated
                        : LLM frozen
    Phase 2 (0.1T tokens) : Alignment
                          : MoonViT + MLP projector updated
                          : LLM frozen
                          : Shift toward instruction-following

Why freeze the LLM in both phases?

MoonViT is being adapted to produce token sequences the LLM can interpret. If the LLM is also updated, it can co-adapt to bad encoder representations rather than forcing the encoder to produce good ones. Freezing the LLM creates pressure on MoonViT to produce high-quality features. (Phase 1 is a CoCa-style stage over ~2T tokens combining a SigLIP contrastive loss with a caption-generation loss; Phase 2 is a ~0.1T-token alignment stage updating only MoonViT and the MLP projector.)⁴⁹

---

7. GNE vs SBE: the design space

quadrantChart
    title Vision Encoder Design Space
    x-axis Low Resolution Support --> High Resolution Support
    y-axis Broken Cross-Region Context --> Preserved Cross-Region Context
    quadrant-1 "Best of both (unsolved)"
    quadrant-2 "GNE: MoonViT, ViT-UHD"
    quadrant-3 "Fixed-res ViT (SigLIP base)"
    quadrant-4 "SBE: LLaVA-UHD, InternVL-2"
    MoonViT: [0.75, 0.85]
    ViT-UHD: [0.80, 0.82]
    SigLIP: [0.25, 0.90]
    LLaVA-UHD: [0.70, 0.35]
    InternVL-2: [0.65, 0.40]

	SBE	GNE (MoonViT)
Cross-region context	✗ broken at boundaries	✓ preserved
Token count	O(N_crops × T_per_crop)	O(H×W/p²) → same, but no redundant CLSs
Attention cost	O(T²) per crop, independent	O(N_total²) — quadratic in full image
Very high-res (>2K)	Tractable by design	Expensive; needs token compression
Global reasoning tasks	Weaker	Stronger

Practice problem: Token budget analysis

Problem: You’re designing an MLLM that processes 2048×2048 screenshots. Budget: max 2048 tokens into the LLM. Using MoonViT’s approach (patch size 14, pixel shuffle 2×2), how many tokens does one image produce? Is the budget met? If not, what compression ratio do you need?

Worked solution:

• Raw patches: $\lceil 2048/14{\rceil }^2\approx 147^2=21609$ tokens
• After 2×2 pixel shuffle: $21609/4\approx 5402$ tokens
• Budget: 2048 tokens — not met, need further $5402/2048\approx 2.6\times$ compression
• Options: another round of pixel shuffle (total 4×), learned token merging (ToMe), or Q-Former style cross-attention down to fixed length
  This is exactly why ViT-UHD and PVC exist — they push beyond pixel shuffle to dynamic compression.

---

8. Reading order

Work through these in order:

1. Start here (architecture): MoonViT-SO-400M on HuggingFace — read the model card and the example code; run the shape walkthrough in your head
2. Code dive: kyegomez/open-moonvit — single-file PyTorch impl; read moonvit.py focusing on MoonViTPatchEmbed, RotaryEmbedding2D, and the cu_seqlens handling
3. Primary source: Kimi-VL Technical Report (arXiv:2504.07491) — Section on MoonViT; focus on the ablation comparing absolute-only vs RoPE-only vs combined positional embeddings
4. Background (if rusty): NaViT paper (arXiv:2307.06304) — the packing strategy; Section 3 is the key part
5. Next frontier: Search “ViT-UHD token compression” — see how the field moved past pixel shuffle

---

9. What’s next

Three threads to pull on next, in suggested order:

1. 2d-rope (not yet written) — 2D RoPE is used beyond MoonViT; understanding the math fully unlocks a lot of modern vision encoder designs
2. siglip (not yet written) — MoonViT is a continued pre-training of SigLIP; understanding SigLIP’s contrastive objective clarifies what MoonViT inherits
3. token-compression (not yet written) — pixel shuffle is just the start; progressive visual compression (PVC), ToMe, and Q-Former are the frontier for making GNE cost-efficient at 2K+ resolution

---

Reference page: moonvit | Filed: 2026-09-02

---

References

Footnotes

1. Zhai et al., “Sigmoid Loss for Language Image Pre-Training (SigLIP),” arXiv:2303.15343 (2023). https://arxiv.org/abs/2303.15343. The SO400M (“shape-optimized,” 400M) variant (27 layers, hidden 1152, patch 14, trained at 224/384) is google/siglip-so400m-patch14-384 (https://huggingface.co/google/siglip-so400m-patch14-384); shape optimization from Alabdulmohsin et al., “Getting ViT in Shape,” arXiv:2305.13035. SigLIP uses learned fixed-size absolute positional embeddings. ↩ ↩²

2. Slice-based / sub-image encoding as used by LLaVA-OneVision (Li et al., “LLaVA-OneVision,” arXiv:2408.03326, https://arxiv.org/abs/2408.03326) and InternVL (Chen et al., “InternVL,” arXiv:2312.14238, https://arxiv.org/abs/2312.14238); the Kimi-VL report contrasts MoonViT’s native-resolution encoding against this sub-image splitting/splicing approach. ↩

3. Dehghani et al., “Patch n’ Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution,” arXiv:2307.06304 (2023). https://arxiv.org/abs/2307.06304 — example/sequence packing that concatenates variable-resolution images into one sequence. ↩

4. Kimi Team, “Kimi-VL Technical Report,” arXiv:2504.07491 (2025). https://arxiv.org/abs/2504.07491 — MoonViT is the 400M native-resolution vision encoder initialized from and continually pre-trained on SigLIP-SO-400M; it adopts NaViT sequence packing, combines interpolated absolute positional embeddings with 2D RoPE (over height and width), feeds an MLP projector with a pixel-shuffle operation, and is trained in a CoCa-style stage (~2T tokens) followed by a ~0.1T-token alignment stage. [recent] ↩ ↩² ↩³ ↩⁴ ↩⁵

5. Dao et al., “FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness,” arXiv:2205.14135 (2022). https://arxiv.org/abs/2205.14135 — the varlen kernel (using cu_seqlens to realize block-diagonal attention without materializing a mask) is provided by the flash-attention library: https://github.com/Dao-AILab/flash-attention. ↩

6. Su et al., “RoFormer: Enhanced Transformer with Rotary Position Embedding,” arXiv:2104.09864 (2021). https://arxiv.org/abs/2104.09864 — rotary position embeddings make the attention dot product depend on relative position; MoonViT uses a 2D factorization across height and width. ↩

7. MoonViT-SO-400M model card and config.json, Moonshot AI. https://huggingface.co/moonshotai/MoonViT-SO-400M — num_hidden_layers 27, hidden_size 1152, patch_size 14, intermediate_size 4304, num_attention_heads 16; ~0.4B params, initialized from SigLIP-SO-400M. ↩

8. Corrected from an earlier draft that claimed SwiGLU. The MoonViT-SO-400M config.json (https://huggingface.co/moonshotai/MoonViT-SO-400M) does not expose an activation field, and MoonViT’s SigLIP-SO-400M base uses gelu_pytorch_tanh — so a GELU-family activation is the supported reading. Confirm against the Kimi-VL report / reference implementation if the exact FFN activation matters. ↩

9. Yu et al., “CoCa: Contrastive Captioners are Image-Text Foundation Models,” arXiv:2205.01917 (2022). https://arxiv.org/abs/2205.01917 — the combined contrastive + captioning training objective that MoonViT’s first training stage follows. ↩