QSA (Qwen Sparse Attention)

Definition

Qwen Sparse Attention (QSA) (Qwen Team, “On the Design of Qwen3.8-Next Architecture,” arXiv:2608.30320; see 2026-09-12-qwen3.8-next) is a natively-trainable sparse attention that scores context at micro-block granularity through a compressed lightweight indexer. Because the indexer scores compressed blocks rather than individual tokens, its cost falls with a compression ratio $r$, taking indexing from $O(n^2)$ down to $O(n^2/r)$. It follows the sparse-attention route of DeepSeek Sparse Attention (DSA, Liu et al. 2025a) but does within-layer block compression instead of cross-layer index sharing.

Key math / How it works

QSA is two stages: a cheap indexer that selects which blocks matter, then a sparse core attention that runs full attention only over the selected blocks.

(A) Compressed lightweight indexer.

• MQA head layout: 4 query heads + 1 shared key head, 128-dim — small enough that indexing is cheap.
• Block compression: keys are average-pooled into blocks of $r$ tokens, ${\hat{k}}_b=\mathrm{RMSNorm}(\mathrm{AvgPool})$, so the indexer scores $\approx n/r$ blocks instead of $n$ tokens.
• Partial RoPE: RoPE on 64 of 128 dims, applied after compression.
• Block importance with block-causal masking:
  $I_{ib}={\sum }_h\mathrm{ReLU}(⟨q_i^h,{\bar{k}}_b⟩).$
• Top-$k_B$ block selection: keep the $k_B=\lceil K/r\rceil$ highest-scoring blocks, expand them back to tokens, and always include the tail of the current block.

(B) Sparse core attention. Full scaled-dot-product attention restricted to the selected blocks’ tokens.

Config: token budget $K=2048$, compression $r=4$ (so $\le 512$ blocks selected).

Native trainability via 2-stage continued pretraining at 256K context:

1. Stage-1 dense distillation — an indexer-only warmup that trains the indexer scores to match a block-maxpooled version of the (dense) teacher attention distribution via a KL loss, while the model still runs dense attention.
2. Stage-2 joint sparse training — the model and indexer train together under the sparse selection.

Why it matters

Trainable sparse attention is a leading 2026 route to cheap long context: rather than approximate a pretrained dense model, the model learns which blocks to attend to end-to-end. QSA’s key move over token-level DSA is compressing the index — dropping indexer cost from $O(n^2)$ to $O(n^2/r)$ — and doing it within-layer, which composes better with hybrid stacks (unlike cross-layer IndexShare). Reported numbers: RULER 512K–1M 93.0 vs 90.1 dense; MRCR 1M 26.4 vs 20.7; at 1M context versus dense GQA, roughly 7.6× prefill / 4.9× decode speedup.

Taught in

• 10-sota-synthesis — QSA as trainable, compressed sparse attention in the 2026 frontier recipe.
• 06-sparse-and-sliding-window — QSA in the sparse-attention taxonomy.

See also

• deepseek-sparse-attention — the trainable-sparse route QSA follows and improves
• sparse-attention — the pattern-masking family it belongs to
• gated-deltanet — the linear-attention half of the Qwen3.8-Next hybrid
• mla — the latent-attention line it contrasts with
• 2026-09-12-qwen3.8-next · attention-glossary · attention-mechanisms