Kimi K3: Open Frontier Intelligence

Authors: Kimi Team (Moonshot AI)
Venue / URL: arXiv:2607.24653v2 (7 Aug 2026)
Ingested: 2026-09-12

Summary

Kimi K3 is a 2.78T-parameter (104.2B active) mixture-of-experts frontier model. Its headline attention contribution is a KDA–MLA hybrid: Kimi Delta Attention — a channel-wise-gated delta-rule linear attention — interleaved 3:1 with MLA, run with NoPE and a 1M-token context. Because KDA carries a constant-size state rather than a growing per-token cache, only the MLA layers hold a (latent-compressed) growing KV cache, so the bulk of the stack is $O(N)$. The report claims ~2.5× scaling efficiency over K2.

Key ideas

• KDA recurrence with channel-wise decay: $S_t=(I-{\beta }_t{k}_t{k}_t^{\top })\mathrm{Diag}({\alpha }_t)S_{t-1}+{\beta }_t{k}_t{v}_t^{\top }$, ${\alpha }_t\in (0,1{)}^{d_k}$; read ${\tilde{o}}_t=S_t^{\top }{q}_t$.
• K3’s two mods over Kimi Linear: (1) lower-bounded decay $g_t=g_{\min }\sigma (e^A{z}_t)\in (g_{\min },0)$, ${\alpha }_t=\exp (g_t)$, $g_{\min }=-5$ — keeps chunkwise rescaling in BF16 range so all tiles use dense Tensor-Core matmuls; (2) full-rank data-dependent output gate $y_t=W_o[\sigma (W_{g}x)\odot \mathrm{RMSNorm}({\tilde{o}}_t)]$.
• Hybrid layout: 3 KDA : 1 Gated-MLA per block + 1 trailing MLA = 93 layers (69 KDA + 24 MLA); NoPE on all MLA layers (KDA supplies position via gating/decay).
• Attention Residuals in the block design.
• KDA context parallelism: fixed-size state all-gather (constant-state makes CP cheap).
• Unified paged cache holding both KDA states and MLA KV.
• MoonViT-V2 vision encoder.

Connections

• kda · gated-deltanet · mla · delta-rule
• moonvit
• 07-linear-and-subquadratic · 10-sota-synthesis

My take

Three tensions worth watching. NoPE viability: relying on KDA’s gating/decay to inject position for the MLA layers is elegant, but it’s an empirical bet — does it hold at 1M, or does length generalization quietly degrade past training length? Constant-state vs recall: the whole economy comes from KDA’s fixed-size state, which is exactly where linear attention historically loses on exact recall; the 24 MLA layers are presumably the recall backstop — the interesting question is whether 24/93 is enough, and whether that ratio is a tuned knob or a fundamental floor. Hybrid ratio: 3:1 KDA:MLA mirrors Qwen3.8-Next’s 3:1 GDN:full-attention — is 3:1 converging as the field’s answer, or coincidence? The bounded-decay-for-BF16-Tensor-Cores trick is the kind of numerics-meets-kernels detail that quietly decides whether a linear-attention layer is actually fast in practice.