Attention Sinks & StreamingLLM

Definition

An attention sink is the empirical phenomenon that a large fraction of attention mass, across layers and heads, is dumped onto the first few tokens of a sequence regardless of their semantic content. StreamingLLM (Xiao et al., 2023; arXiv:2309.17453) exploits this to enable stable, constant-memory decoding over unbounded streams: keep a few initial sink tokens permanently in the KV cache alongside a rolling recent window.

Key math / How it works

Why sinks exist. Softmax is a convex combination — weights are non-negative and must sum to 1:

${\alpha }_{ij}=\frac{e^{s_{ij}}}{{\sum }_k{e}^{s_{ik}}},{\sum }_j{\alpha }_{ij}=1.$

A query has no way to express “attend to nothing,” so even when the right answer is no-op it must park its leftover mass somewhere. The network learns a dumping ground, and the first tokens are the natural choice: under causal masking they are the only positions visible to every query — the universally reachable “ground” of the causal graph.

The failure a naive fix hits. For infinite streaming with bounded memory, the obvious move is a rolling KV window (keep the last $w$ tokens, evict the oldest). This catastrophically fails — perplexity spikes the moment the sink tokens are evicted, because queries must redistribute their large parked mass onto tokens never trained to receive it, pushing distributions out of distribution.

StreamingLLM. Keep the first $k$ tokens (Xiao et al. find $k\approx 4$ suffices) plus the recent window $w$, dropping the middle. Cache is $O(k+w)$, constant. The dumping ground persists, distributions stay in-distribution, and the model decodes stably over millions of tokens with no fine-tuning. Related proposed fix: a learned “off” logit / +1 in the softmax denominator (softmax-off-by-one, Evan Miller) gives an explicit zero-sink slot [verify: this "+1" fix is Miller's blog post, not the StreamingLLM paper].

Why it matters

Sinks are the “softmax must sum to 1” tax made visible, and StreamingLLM makes it actionable: fixed-window models can generate coherently forever at constant memory. Caveat — it extends stability, not memory: evicted middle tokens are gone, so it does not grant recall of far-back content. Modern models increasingly bake in explicit sink tokens to make windowed KV eviction robust.

Taught in

• 06-sparse-and-sliding-window — §4 the sink phenomenon and the StreamingLLM fix.

See also

• sliding-window-attention — the rolling window it augments
• sparse-attention · kv-cache · online-softmax
• attention-glossary · attention-mechanisms