MoonViT

Native-resolution vision encoder from Kimi-VL (Moonshot AI, April 2025). Initialized from SigLIP-SO-400M (~400M params) and continued pre-trained to process arbitrary-resolution images in a single global forward pass. The canonical example of Global Native-Resolution Encoding (GNE).

Why it exists

Standard ViTs process fixed-resolution images. In MLLMs, two naive approaches fail:

• Resize → loses fine-grained detail (OCR, small objects, dense charts)
• Slice-Based Encoding (SBE) → breaks cross-region context at tile boundaries; redundant CLS tokens

MoonViT encodes the full image at native resolution in one pass.

Architecture

image (H, W, 3)
    ↓
MoonViTPatchEmbed          # Conv2d(3, 1152, kernel=14, stride=14)
    ↓
AbsolutePosEmbedInterpolator  # bicubic interp of SigLIP learned embeds → (H/14, W/14)
    ↓
[×27] MoonViTEncoderLayer
    │  ├─ LayerNorm
    │  ├─ Multi-head Self-Attention (flash_attn_varlen / block-diag SDPA)
    │  │   └─ + RotaryEmbedding2D on Q, K
    │  ├─ Residual
    │  ├─ LayerNorm
    │  └─ SwiGLU MLP → Residual
    ↓
LayerNorm → (N_tokens, B, 1152)
    ↓
MLPProjector
    ├─ 2×2 pixel-shuffle (4× token reduction, dim 1152 → 4608)
    └─ Linear → GELU → Linear (→ LLM dim)

Key specs: 27 layers, hidden dim 1152, patch size 14px, bf16.

Key mechanisms

NaViT sequence packing

Images of varying sizes are concatenated into one flat sequence rather than padded. cu_seqlens marks image boundaries. FlashAttention’s flash_attn_varlen_func enforces block-diagonal masking so tokens only attend within their image. No cross-image leakage, no padding waste.

${\text{Attention}}_i=\text{softmax}\left(\frac{Q_i{K}_i^{\top }}{\sqrt{d}}\right)V_i$

where subscript $i$ selects tokens from image $i$ only.

Dual positional embeddings

Neither absolute nor RoPE alone is sufficient:

• Interpolated absolute embeds: preserves SigLIP pretraining signal; bicubically interpolated to $(H/14,W/14)$ grid at inference
• 2D RoPE: encodes relative positions multiplicatively in attention ($q^{\top }{R}_{m-n}k$); robust to unseen resolutions

Together they let MoonViT generalize to native resolution without retraining.

MLP projector (pixel shuffle)

2×2 space-to-depth collapses each 2×2 patch-token block into one token (4× reduction). Keeps token count tractable for the LLM while preserving local structure.

Training

Phase 1 — Continued pre-training (2T tokens)

• SigLIP contrastive loss + caption generation (CoCa-style)
• MoonViT + text decoder updated; LLM frozen

Phase 2 — Alignment (0.1T tokens)

• MoonViT + MLP projector updated; LLM frozen
• Shifts distribution toward instruction-following

GNE vs SBE

	SBE (e.g. InternVL-2, LLaVA-UHD)	GNE (MoonViT)
Cross-region context	Broken at boundaries	Preserved
Attention cost	O(T²) per crop, independent	O(N_total²) — quadratic in full image
Resolution flex	Easy	Harder at very high res
Global reasoning	Weaker	Stronger

GNE’s main cost: at 1920×1080, $p=14$ → ~9800 tokens → ~96M attention ops per image. FlashAttention makes it tractable; token compression is the frontier.

Benchmarks (Kimi-VL-A3B)

Benchmark set avg	TTFT
66.9% (MMBench, SEED-Img, AI2D, MMStar, SQA, HallusionBench)	~296 ms

ViT-UHD surpasses it: 67.5% avg, 121 ms TTFT — via more aggressive token compression.

Limitations

• Quadratic attention cost scales badly with resolution (partially mitigated by FlashAttention)
• Pixel shuffle is a crude token compressor; learnable compression (PVC, token merging) is better
• No explicit token reduction before the LLM projector

See also

• global-native-encoding — the broader paradigm
• slice-based-encoding — the alternative approach
• navit-packing — the sequence packing strategy
• 2d-rope — rotary position embeddings for 2D grids
• siglip — the foundation model MoonViT initializes from
• moonshot-ai — the lab behind Kimi-VL
• multimodal-llms — broader context

Resources

• Kimi-VL Technical Report: arXiv:2504.07491
• HuggingFace weights: moonshotai/MoonViT-SO-400M
• Open PyTorch impl: kyegomez/open-moonvit