MoBA (Mixture of Block Attention)

Definition

MoBA (Mixture of Block Attention) (Moonshot AI, “MoBA: Mixture of Block Attention for Long-Context LLMs”, arXiv:2502.13189 [arxiv-verified], Feb 2025) applies the Mixture-of-Experts idea to attention: the context is partitioned into fixed-size blocks, a parameter-less gate routes each query to only its top-$k$ most relevant blocks, and attention runs over the union of selected blocks. It is a trainable block-sparse attention that can switch between full and sparse behavior, requires continued training to adopt, and is deployed in Kimi for long context.

Key math / How it works

Split the context into blocks of size $B$; let $I_i$ index the tokens of block $i$. The per-block affinity between a query $q$ and block $i$ is the inner product of $q$ with the mean-pooled keys of that block:
$s_i=⟨q,\mathrm{meanpool}(K[I_i])⟩$
The gate keeps a block iff its score is among the top-$k$:
$g_i=1[s_i\in \mathrm{TopK}(\{s_j\},k)]$
and attention is computed only over the selected tokens $I={\bigcup }_{i:g_i=1}{I}_i$:
$\mathrm{out}=\mathrm{Softmax}(qK[I{]}^{\top })V[I]$
Causality is enforced two ways: future blocks are masked by setting $s_i=-\infty$ so they can never be selected, and the current block is always selected and attended with a standard within-block causal mask. The gate has no learned parameters (it reuses $q$ and mean-pooled $K$), so MoBA drops in with continued training rather than a from-scratch rewrite, and the same weights can run either full dense attention or sparse block attention.

Why it matters

MoBA reframes long-context sparsity as routing — the same top-$k$ discipline that makes MoE FFNs cheap, now applied to which slabs of context a query even looks at. Because the gate is parameter-less and the model can fall back to full attention, it is a low-risk retrofit for an already-trained model, which is why it shipped in Kimi’s long-context stack. It sits alongside DSA (token-level indexer) and QSA (compressed block indexer) as one of the three main trainable-sparse routes of 2026, distinguished by its explicit MoE-style block gating.

Taught in

• 06-sparse-and-sliding-window — MoBA as block-sparse top-$k$ routing.
• 10-sota-synthesis — trainable-sparse routes in the frontier synthesis.

See also

• deepseek-sparse-attention — token-level lightning-indexer top-$k$
• qwen-sparse-attention — compressed block-granularity indexer
• sparse-attention — the family it belongs to
• kda — the Kimi delta-rule linear-attention line