NaViT Sequence Packing

Training and inference strategy from NaViT (Google, 2023) for processing variable-resolution images in a ViT without padding waste. Used by MoonViT and increasingly common in production vision encoders.

How it works

Instead of padding all images in a batch to the same size:

1. Patchify each image independently → variable-length token sequences
2. Concatenate all sequences end-to-end into one flat 1D tensor
3. Record cumulative lengths in cu_seqlens (shape: [B+1], values: [0, N_1, N_1+N_2, ...])
4. Run attention with block-diagonal masking: token $i$ attends only to tokens from the same image

Why it’s efficient

• No compute spent on padding tokens
• FlashAttention’s flash_attn_varlen_func natively supports this via cu_seqlens — no explicit masking matrix needed
• Packs more real signal per GPU memory allocation

Attention masking

The effective attention pattern is block-diagonal:

$A_{ij}=\{\begin{array}{ll}\text{softmax}(q_i^{\top }{k}_j/\sqrt{d})& \text{if }i,j\text{ belong to same image}\\ -\infty & \text{otherwise}\end{array}$

See also

• moonvit
• global-native-encoding