KDA (Kimi Delta Attention)

Definition

Kimi Delta Attention (KDA) is a gated delta-rule linear attention with a channel-wise forget gate: the decay ${\alpha }_t$ is a vector in $(0,1{)}^{d_k}$ rather than the single scalar of Gated DeltaNet, so each state channel can retain or forget at its own rate. KDA originates in Kimi Linear (arXiv:2510.26692) and is the linear-attention half of the KDA–MLA hybrid in the Kimi K3 technical report (Kimi Team / Moonshot AI, arXiv:2607.24653; see 2026-09-12-kimi-k3). It keeps a constant-size state per head — not a growing KV cache — giving $O(N)$ compute and cheap million-token context.

Key math / How it works

The core recurrence (Kimi Linear Eq. 1), per head with state $S\in {\mathbb{R}}^{d_k\times d_v}$:

$S_t=(I-{\beta }_t{k}_t{k}_t^{\top })\mathrm{Diag}({\alpha }_t)S_{t-1}+{\beta }_t{k}_t{v}_t^{\top },{\tilde{o}}_t=S_t^{\top }{q}_t.$

Versus GDN’s scalar ${\alpha }_t$, the $\mathrm{Diag}({\alpha }_t)$ with ${\alpha }_t\in (0,1{)}^{d_k}$ applies a per-channel decay to the state before the delta write.

Parameterization (per token $x$):

$q,k=L2Norm(\mathrm{Swish}(\mathrm{ShortConv}(Wx))),v=\mathrm{Swish}(\mathrm{ShortConv}(Wx)),$
$\beta =\sigma (W_{\beta }x),z_t=W_{\alpha \uparrow }{W}_{\alpha \downarrow }x+b_{\alpha }(\text{low-rank decay logits}).$

K3’s two modifications over Kimi Linear:

1. Lower-bounded decay. Instead of an unbounded decay, K3 uses a scaled sigmoid
   $g_t=g_{\min }\cdot \sigma \left(e^A{z}_t\right)\in (g_{\min },0),{\alpha }_t=\exp (g_t),g_{\min }=-5.$
   Bounding $g_t$ from below keeps the chunkwise rescaling factors in the BF16 dynamic range, which lets every tile use dense Tensor-Core matmuls (no per-tile fallback for numerically extreme decays).
2. Full-rank data-dependent output gate. The read-out is gated and normalized:
   $y_t=W_o[\sigma (W_{g}x)\odot \mathrm{RMSNorm}({\tilde{o}}_t)].$
   Unlike a low-rank gate, this is full-rank and data-dependent.

Training uses the delta-rule chunkwise-parallel form — recurrent across chunks, matmul-parallel within a chunk — so KDA trains at high throughput while inference remains a constant-state recurrence.

Hybrid in K3. Blocks interleave 3 KDA : 1 Gated-MLA, plus 1 trailing MLA layer, over 93 layers total = 69 KDA + 24 MLA. All MLA layers use NoPE (no positional encoding) — KDA supplies position implicitly through its gating/decay dynamics. Because KDA carries a fixed-size state rather than a per-token KV cache, only the MLA layers keep a (latent-compressed) growing cache; the KDA layers cost $O(N)$ and are what make 1M context affordable.

Why it matters

KDA sits at the KV-cache / linear-attention frontier as of 2026. By making decay channel-wise it recovers more of softmax attention’s selectivity than scalar-gated GDN, and the bounded-decay trick makes the chunkwise kernel fully Tensor-Core-dense in BF16. Combined with a sparse MLA backbone and NoPE, the hybrid delivers frontier quality with near-constant memory growth — making 1M-token context economical rather than aspirational.

Taught in

• 07-linear-and-subquadratic — KDA as channel-wise-gated delta rule; the recurrence and chunkwise form.
• 10-sota-synthesis — the KDA–MLA hybrid as a 2026 frontier recipe.

See also

• gated-deltanet — the scalar-decay predecessor KDA generalizes
• delta-rule — the error-correcting write at its core
• mla — the full-attention half of the K3 hybrid
• linear-attention · kv-cache — the baselines it displaces
• 2026-09-12-kimi-k3 · attention-glossary · attention-mechanisms