Delta Rule (DeltaNet)

Definition

The delta rule is a fast-weight update for linear attention that, before writing a new key–value association into the recurrent state, first removes the value currently associated with that key — an error-correcting write rather than a blind accumulation. Where additive linear attention does $S_t=S_{t-1}+k_t{v}_t^{\top }$ (every key ever seen keeps piling onto the state), DeltaNet treats $S$ as an associative memory and moves it toward the target association by a step ${\beta }_t$, correcting the memory’s current prediction error at $k_t$. This is the classic Widrow–Hoff delta rule reused as an online write rule for a matrix-valued state.

Key math / How it works

Let the state be $S\in {\mathbb{R}}^{d_k\times d_v}$. The current read-out (prediction) at key $k_t$ is ${\hat{v}}_t=S_{t-1}^{\top }{k}_t$. The delta rule nudges the state to reduce the error $v_t-{\hat{v}}_t$:

$S_t=S_{t-1}+{\beta }_t{k}_t(v_t-S_{t-1}^{\top }{k}_t{)}^{\top }.$

Expanding the correction term gives the equivalent linear-recurrence form,

$S_t=(I-{\beta }_t{k}_t{k}_t^{\top })S_{t-1}+{\beta }_t{k}_t{v}_t^{\top },$

so each step applies a rank-1 removal $(I-{\beta }_t{k}_t{k}_t^{\top })$ that erases the component of the old state aligned with $k_t$, then writes the fresh association ${\beta }_t{k}_t{v}_t^{\top }$. With ${\beta }_t\in (0,1)$ a data-dependent write strength: ${\beta }_t\to 0$ leaves the memory untouched, ${\beta }_t\to 1$ overwrites the slot at $k_t$ completely. Read-out is the usual linear-attention query $y_t=S_t^{\top }{q}_t$.

Lineage. The delta rule as a fast-weight programmer is due to Schlag, Irie & Schmidhuber, “Linear Transformers Are Secretly Fast Weight Programmers” — which reframes linear attention as a network that writes to and reads from a fast-weight matrix, and swaps the additive write for the delta write. Yang et al., “Parallelizing Linear Transformers with the Delta Rule over Sequence Length,” made it trainable at scale: the sequential rank-1 updates are recast into a chunkwise-parallel form (a WY-style representation of the product of $(I-{\beta }_t{k}_t{k}_t^{\top })$ transforms) so training runs as matmuls over chunks while inference stays a constant-size recurrence.

Why it matters

Additive linear attention has no mechanism to forget: distinct keys that are not orthogonal collide in the fixed-size state, and the collisions accumulate without bound as the sequence grows — the mechanistic root of linear attention’s recall gap. The delta rule fixes exactly this by overwriting a key’s slot instead of superposing onto it, giving a genuinely error-correcting associative memory at the same $O(N)$ compute and constant state. This is the primitive underneath the 2024–26 delta-rule wave — Gated DeltaNet and Kimi Delta Attention — which add data-dependent decay on top of the delta write.

Taught in

• 07-linear-and-subquadratic — the delta rule as the error-correcting fix to additive linear attention.

See also

• linear-attention — the additive baseline whose key collisions the delta rule corrects
• gated-deltanet — delta write + data-dependent scalar decay
• kda — delta write + channel-wise decay
• state-space-models — the recurrent-state cousin line
• attention-glossary · attention-mechanisms