Local/Global Interleaved Attention

Definition

Local/global interleaved attention alternates cheap local layers (bounded sliding-window attention) with sparse, expensive global (full-context) layers, so that only the global layers hold a full-context KV cache. It is the defining recipe of the Gemma line and the simplest, most widely-shipped form of KV-cache control — complementary to MLA and trainable-sparse routes rather than competing with them.

Key math / How it works

Two knobs dominate: the local:global ratio (how many bounded-window layers per full-context layer) and the window size of the local layers. Across the Gemma generations (all [arxiv-verified]):

• Gemma 2 (arXiv:2408.00118): ratio 1:1, sliding window 4096, attention-logit soft-capping (50 for global / 30 for local), GQA.
• Gemma 3 (arXiv:2503.19786): ratio 5:1, window shrunk to 1024, per-layer-type RoPE base (local 10k / global 1M), and QK-norm replaces soft-capping. Result: KV-cache overhead < 15% at 32K context, versus ~60% for a global-only model.
• Gemma 4 (arXiv:2607.02770): ratio 5:1 (4:1 for the E2B variant), pp-RoPE ($p=0.25$) on the global layers → −37.5% global KV cache, a “values = keys” reuse trick in global layers, plus MoE and audio.

The mechanism: a query in a local layer attends only within its window $[t-w,t]$, so that layer’s cache is bounded by $w$ regardless of sequence length; only the sparse global layers attend to the full prefix and therefore pay the $O(n)$ cache. Shrinking $w$, raising the local:global ratio, and giving local/global layers different RoPE bases (short-range vs. long-range) jointly drive the full-context cache toward a small fraction of total.

Why it matters

It is the simplest widely-shipped KV-cache lever: no new attention kernel, no learned selector — just interleave bounded-window layers and reserve full attention for a minority of layers. Because it acts at the layer-scheduling level, it stacks cleanly with orthogonal savings (MLA’s latent cache, DSA-style token selection, quantized KV). The Gemma trajectory — window $4096\to 1024$, ratio $1:1\to 5:1$, split RoPE bases, pp-RoPE and value/key reuse — shows the design being squeezed generation over generation for ever-cheaper long context.

Taught in

• 06-sparse-and-sliding-window — interleaved local/global as bounded-window sparsity.
• 10-sota-synthesis — the Gemma cache-control recipe in the frontier synthesis.

See also

• sliding-window-attention — the bounded-window local layer
• rope-scaling-yarn — per-layer RoPE base and long-context scaling
• kv-cache — the cost this recipe controls
• gqa-mqa — the head-sharing used alongside it