Gated DeltaNet (GDN)

Definition

Gated DeltaNet (Yang, Kautz & Hatamizadeh, “Gated Delta Networks: Improving Mamba2 with Delta Rule,” arXiv:2412.06464 [verify]) is a linear-attention primitive that combines two forgetting mechanisms that were previously used separately: Mamba-2-style data-dependent scalar decay (a global “how long should the whole state live” gate) and the delta rule (a targeted “overwrite this key’s slot” write). Plain delta-rule DeltaNet can correct a key’s association but cannot let the state as a whole fade; Mamba-2 can decay the state but writes additively and so still suffers key collisions. GDN gets both — selective decay and error-correcting writes — in one constant-size recurrence.

Key math / How it works

Per head, with state $S\in {\mathbb{R}}^{d_k\times d_v}$:

$S_t={\alpha }_t(I-{\beta }_t{k}_t{k}_t^{\top })S_{t-1}+{\beta }_t{k}_t{v}_t^{\top },y_t=S_t^{\top }{q}_t.$

Two data-dependent gates control it:

• ${\alpha }_t\in (0,1)$ — scalar decay applied to the whole state, setting its effective lifetime (the Mamba-2 contribution). ${\alpha }_t\to 1$ retains, ${\alpha }_t\to 0$ flushes.
• ${\beta }_t\in (0,1)$ — delta write strength on the rank-1 removal-and-write $(I-{\beta }_t{k}_t{k}_t^{\top })S_{t-1}+{\beta }_t{k}_t{v}_t^{\top }$ (the DeltaNet contribution).

Setting ${\alpha }_t\equiv 1$ recovers plain DeltaNet; setting ${\beta }_t{k}_t{v}_t^{\top }$ to a pure additive write recovers Mamba-2 — GDN is the strict superset.

Parameterization (per token input $x$):

$q,k=L2Norm(\mathrm{SiLU}(\mathrm{ShortConv}(Wx))),v=\mathrm{SiLU}(\mathrm{ShortConv}(Wx)),$
$\beta =\sigma (W_{\beta }x),\alpha =\exp (-\exp (A)\cdot \mathrm{softplus}(W_{\alpha }x+b)).$

The short depthwise conv gives local mixing, L2-normalized $q,k$ stabilize the delta write, and the $\alpha$ parameterization keeps the decay strictly in $(0,1)$. Training uses the DeltaNet chunkwise-parallel form (recurrent across chunks, matmul-parallel within a chunk), so GDN trains at Transformer-like throughput while inference is a constant-state recurrence.

Why it matters

GDN is the leading modern linear-attention primitive: it closes much of plain linear attention’s recall gap (delta write) while retaining a controllable state lifetime (decay), all at $O(N)$ compute and constant per-token state. It is deployed as the linear-attention layer in the Qwen3.8-Next hybrid, interleaved 3 GDN : 1 full attention, and it is the direct basis for Kimi Delta Attention, which generalizes the scalar decay ${\alpha }_t$ to a per-channel vector.

Taught in

• 07-linear-and-subquadratic — GDN as delta rule + Mamba-2 decay; the modern linear-attention layer.

See also

• delta-rule — the error-correcting write GDN gates
• kda — GDN with channel-wise (vector) decay
• mamba — the data-dependent scalar decay GDN borrows
• linear-attention — the additive baseline
• attention-glossary · attention-mechanisms