Sliding-Window Attention

Definition

Sliding-window (local) attention restricts each token to attend only to itself and the previous $w$ tokens, masking everything else before softmax. Popularized at scale by Mistral 7B (Jiang et al., 2023; arXiv:2310.06825), it drops the attention core from $O(s^2)$ to $O(s\cdot w)$ — linear in sequence length — and bounds the decode-time KV cache at $O(w)$ via a rolling buffer. Global reach is recovered through depth.

Key math / How it works

The causal window mask is

$M_{ij}=1[i-w<j\le i],A=\mathrm{softmax}(\frac{Q{K}^{\top }}{\sqrt{d_k}}+\log M)V.$

Allowed entries per row $\le w$, so core FLOPs are $\approx 4bswd$ instead of $4b{s}^{2}d$.

Effective receptive field via depth (the CNN analogy). Layer 1 lets token $i$ see back to $i-w$; those tokens already pulled from their windows, so after $\ell$ layers $i$ is influenced by tokens as far as $i-\ell w$. An $L$-layer stack thus reaches

$\text{effective receptive field}\approx L\cdot w.$

Mistral: $w=4096$, $L=32$ → $L\cdot w=131,072$ tokens of theoretical reach, far beyond any single layer’s 4096 window.

Decode memory. Only the last $w$ tokens’ K/V are ever needed, so the cache is bounded at $O(w)$ (Mistral’s fixed-size rolling buffer) rather than growing $O(s)$. Switching a 32k layer to $w=2048$: $Q{K}^{\top }$ scores $\times w/s=1/16$, and the cache $16\times$ smaller and constant as generation continues.

Why it matters

Sliding-window is the cheap workhorse long-context primitive: no new parameters, just a masked SDPA and a rolling KV buffer. Its failure mode is precise — dependencies farther than $L\cdot w$ cannot be bridged — which is exactly why production “hybrid” stacks interleave a few full-attention (or global-token) layers among mostly-local ones for one-hop long-range reach. It underlies both training-time locality (Mistral) and inference-time streaming (see StreamingLLM).

Taught in

• 06-sparse-and-sliding-window — §3a window mask, receptive field, rolling cache.

See also

• sparse-attention — window + global + random patterns
• attention-sinks — sinks + rolling window for infinite streaming
• kv-cache · flash-attention — windowed masks in fused kernels
• attention-glossary · attention-mechanisms