DSA (DeepSeek Sparse Attention)

Definition

DeepSeek Sparse Attention (DSA) is a natively-trainable sparse attention that uses a lightweight lightning indexer to select, per query token, a top-$k$ subset of context, then runs full attention only over that subset. It was introduced in DeepSeek-V3.2-Exp (2025; tech report “DeepSeek-V3.2-Exp: Boosting Long-Context Efficiency with DeepSeek Sparse Attention”, distributed as a GitHub PDF — no arXiv ID [verify]). DSA composes on top of MLA: MLA decides how the KV is stored (latent compression), DSA decides which tokens are attended. It is the head of a lineage — DSA → CSA → CSA2 — that has become the dominant trainable-sparse route among frontier labs, and has been adopted cross-lab (e.g. GLM-5, model tag GLM_MOE_DSA).

Key math / How it works

Lightning indexer. For query position $t$ and candidate context position $s$, the indexer computes an index score over indexer heads (with low-rank query compression):
$I_{t,s}={\sum }_h{w}_{t,h}^I\mathrm{ReLU}(q_{t,h}^I\cdot k_s^I)$
The indexer is deliberately cheap (few heads, compressed queries, ReLU rather than softmax). Token-level top-$k$ selection then keeps the highest-scoring context tokens, and only that selected KV is fed into the (MLA) attention. The exact top-$k$ budget and trained context length are [verify] (in the unparsed PDF). DeepSeek open-sourced the kernels (TileLang reference; DeepGEMM/FlashMLA) and reports “virtually identical output quality” versus the dense V3.1-Terminus baseline.

CSA (Compressed Sparse Attention) — the successor in DeepSeek-V4 (arXiv:2606.19348 [arxiv-verified], “DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence”). CSA first compresses the KV by a factor $1/m$ (every $m$ tokens collapse to one entry via two learnable streams), then applies DSA-style top-$k$ selection over the compressed entries. It adds HCA (Heavily Compressed Attention), which consolidates distant context at a larger ratio $m^{\prime }$ with no selection at all. Reported cost: ~27% of per-token inference FLOPs and ~10% of the KV cache versus V3.2 at 1M-token context [verify numbers]; the model also uses mHC and the Muon optimizer.

CSA2 — DeepSeek-V4.1-Flash (repo PDF “Pushing the Limits of KV Cache Compression”; no arXiv [verify]). Per-layer static modes — Full / Reindex / Reuse (Reuse shares KV and indexer keys across layers) — plus a Hierarchical Sparse Indexer. With FP4 KV, footprint is ≈ 890 bytes/token [verify].

Why it matters

DSA is the dominant trainable-sparse route of the 2026 frontier: rather than approximating a dense pretrained model post-hoc, the sparse selection is trained natively and stacks cleanly on MLA’s already-compressed latent cache. The lineage keeps attacking the two long-context walls — selection compute (DSA) and KV-cache size (CSA compression, CSA2’s FP4 + layer sharing). Cross-lab adoption (GLM-5’s GLM_MOE_DSA) is independent corroboration that the lightning-indexer + top-$k$ recipe generalizes beyond DeepSeek.

Taught in

• 10-sota-synthesis — DSA/CSA as the trainable-sparse-on-MLA route in the frontier synthesis.
• 06-sparse-and-sliding-window — token-level top-$k$ selection versus block-sparse alternatives.

See also

• qwen-sparse-attention — the block-compressed indexer variant ($O(n^2)\to O(n^2/r)$)
• mla — the latent attention DSA is layered on
• sparse-attention — the family it belongs to
• moba — MoE-style block-selection sparse attention
• attention-mechanisms