On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability

Authors: Qwen Team (Alibaba)
Venue / URL: arXiv:2608.30320v1 (31 Aug 2026)
Ingested: 2026-09-12

Summary

Qwen3.8-Next is a sparse-MoE model (125B total / 6B active, plus a 51B off-accelerator n-gram embedding) that the report claims matches a 397B-A17B flagship at roughly 1/3 the activated params and ~1/9 the training FLOPs. Four design pillars: a GDN + full-attention hybrid (3:1), QSA (compressed trainable sparse attention), a Gated Residual, and an n-gram embedding. Trained with the Muon optimizer.

Key ideas

• GDN recurrence: $S_t={\alpha }_t(I-{\beta }_t{k}_t{k}_t^{\top })S_{t-1}+{\beta }_t{k}_t{v}_t^{\top }$, $y_t=S_t^{\top }{q}_t$; scalar data-dependent decay ${\alpha }_t$ + delta write ${\beta }_t$. Used as the linear-attention layer, 3 GDN : 1 full attention.
• QSA compressed indexer: micro-block scoring via a compressed lightweight indexer, dropping indexing from $O(n^2)$ to $O(n^2/r)$; MQA (4 query + 1 shared key head, 128-dim), keys avg-pooled into blocks of $r$, partial RoPE (64/128) after compression, $I_{ib}={\sum }_h\mathrm{ReLU}(⟨q_i^h,{\bar{k}}_b⟩)$, top-$k_B=\lceil K/r\rceil$ blocks + current-block tail. Config $K=2048$, $r=4$ ($\le 512$ blocks).
• Native trainability: 2-stage continued pretraining at 256K — Stage-1 dense distillation (KL of indexer scores to block-maxpooled teacher attention, indexer-only warmup), Stage-2 joint sparse training.
• Long-context gains: RULER 512K–1M 93.0 vs 90.1 dense; MRCR 1M 26.4 vs 20.7; at 1M vs dense GQA, ~7.6× prefill / ~4.9× decode.
• Positioning: follows/improves DeepSeek Sparse Attention (DSA, Liu et al. 2025a); within-layer compression vs cross-layer IndexShare (better for hybrids).
• Methodology note: “loss vs downstream accuracy disagree” — evaluates design choices on downstream accuracy, not just LM loss.

Connections

• qwen-sparse-attention · gated-deltanet · deepseek-sparse-attention · mla
• 06-sparse-and-sliding-window · 10-sota-synthesis

My take

Worth flagging up front: this is Qwen (Alibaba), not DeepSeek — but it explicitly builds on DeepSeek’s DSA route, which is a nice example of how fast the trainable-sparse idea has propagated across labs. The more interesting signal is the recipe convergence: a hybrid linear layer (GDN) + trainable sparse full-attention (QSA) looks like the emerging frontier template — and the 3:1 linear:full ratio here matches Kimi K3’s 3:1 KDA:MLA. The two labs picked different linear primitives (scalar-gated GDN vs channel-wise KDA) and different long-context strategies (trainable-sparse full attention vs MLA + NoPE), but land on the same skeleton. The “loss vs downstream accuracy disagree” methodological point is a quiet but important caveat for anyone ablating these designs on LM loss alone.