Long Context

Definition

Long context is the problem of training a model at one sequence length (say 4k) and serving it at a much larger one (128k, 1M). It is really two orthogonal problems wearing one name: (1) positional generalization — RoPE angles go out-of-distribution at unseen positions — and (2) compute & memory — attention is $O(s^2)$ in FLOPs and the KV cache is $O(s)$ in bytes. Shipping real long context means stacking a positional-extension recipe on top of a systems recipe, then measuring whether the model actually uses the window.

Key math / How it works

Two scalings define the race (let $s$=seq len, $d$=model dim, $L$=layers, $n_{kv}$=KV heads, $b$=batch, $p$=bytes):

Quantity	Scaling	At long $s$
Attention-core FLOPs (prefill)	$\Theta (s^{2}d)$ per layer	quadratic — dominates compute
KV cache	$2L{n}_{kv}{d}_{h}sbp$ bytes	linear, but read every decode step
Score-matrix HBM (naive)	$\Theta (s^2)$	the wall FlashAttention removes

The four levers:

• Positions — Position Interpolation / NTK-aware / YaRN keep RoPE angles in range. Nearly free in FLOPs; makes the model coherent far out.
• Prefill compute/storage — FlashAttention tiles with online softmax so the $s\times s$ scores are never materialized ($\Theta (s)$ traffic).
• Distribution — Ring Attention / context parallelism shard the sequence so context scales with device count.
• Decode affordability — GQA / MLA shrink the KV cache so a long sequence fits and each decode step reads less.

Worked scale (Llama-3-70B-style, $s$=128k): KV cache $\approx 40$ GiB with GQA ($g=8$) — vs. $\sim$320 GiB with plain MHA, physically impossible on one GPU. Attention-core prefill $\approx 4.5\times 10^{16}$ FLOPs, which has overtaken the entire rest of the model ($\approx 1.8\times 10^{16}$). Prefill is compute-bound and quadratic; decode is memory-bound on the cache.

Advertised ≠ usable. A model can be extended to a huge context and still not reason over it. Evaluation:

• Perplexity is insufficient — dominated by local next-token prediction; hides whether distant info is used.
• Needle-in-a-haystack (NIAH) — plant a fact at controlled depth × length; a big step up, but only single-fact verbatim retrieval (often shows a weak middle, “lost in the middle”).
• RULER (arXiv:2404.06654 [verify]) — generalizes NIAH to multi-key/multi-hop tracking, aggregation, and QA, probing reasoning. Headline: effective context length is usually far below the advertised one.

Why it matters

“1M-token” is a claim about the maximum sequence the code accepts, not a guarantee of reasoning over that span. Positional extension makes it coherent, FlashAttention/Ring make it runnable, GQA/MLA make it affordable — but only RULER-style aggregation tasks reveal the usable budget, which is often a small fraction of the number on the box. Treat vendor context numbers as upper bounds. High-resolution images and video (vlm-fusion) reintroduce the same $s^2$/KV pressures on the vision side.

Taught in

• 09-long-context-race — the full two-problem framing, systems levers, and evaluation.

See also

• rope-scaling-yarn — the positional-extension half
• ring-attention · context-parallelism — sequence sharding
• flash-attention — removes the $s^2$ storage/traffic wall
• kv-cache · gqa-mqa · mla — the affordability floor
• sliding-window-attention · sparse-attention · linear-attention — the other bet: change the algorithm
• attention-mechanisms