Ring Attention

Definition

Ring Attention (Liu, Zaharia, Abbeel, 2023; arXiv:2310.01889) computes exact attention over sequences far larger than one device’s memory by sharding the sequence across $N$ devices arranged in a ring and rotating K/V blocks around it, overlapping the block transfers with compute. Per-device activation memory drops to $O(s/N)$, so maximum context scales linearly with device count — “near-infinite” context becomes a matter of adding GPUs. It is not an approximation; it is blockwise online-softmax attention distributed across the sequence dimension.

Key math / How it works

Device $j$ holds query block $Q_j$ (resident) and initially its local $K_j,V_j$. Each step:

1. Compute the blockwise, online-softmax attention of local $Q_j$ against the K/V block it currently holds, updating running softmax state $(m,\ell ,\mathbf{o})$ — the $s\times s$ score matrix is never materialized anywhere; each device only forms the small $Q_j\times (\text{current K/V block})$ tile on-chip.
2. Send its current K/V block to its ring neighbor and receive the next one.

After $N$ steps every query has attended to every key → exact attention.

The overlap trick. The P2P send/recv of the next K/V block runs concurrently with compute on the current block. Communication is fully hidden — near-zero wall-clock overhead — when

$t_{\text{compute per block}}\ge t_{\text{comm per block}},$

which holds once the block size is large enough. Memory per device is $O(s/N)$ activations, so doubling devices roughly doubles the affordable context.

Why it matters

FlashAttention fits attention on one device up to its memory; Ring Attention removes that ceiling, making million-token context a hardware-scaling question rather than an algorithmic wall. It is exactly context parallelism (CP) in the parallelism taxonomy — the mechanism behind splitting the sequence dimension for attention (distinct from sequence parallelism, which splits the LayerNorm/residual/dropout regions). CP composes with DP × TP × PP; because its P2P traffic is on the critical path, the ring lives on a fast interconnect. Note it solves only the systems half of long context — the RoPE angles must still be in range (rope-scaling-yarn) for output to be coherent.

Taught in

• 09-long-context-race — §4, the ring rotation + compute/comm overlap, and the tie to context parallelism.

See also

• flash-attention · online-softmax — the blockwise recurrence Ring builds on
• context-parallelism — Ring Attention is CP; sequence-parallelism for contrast
• long-context — the umbrella problem
• rope-scaling-yarn — the orthogonal positional-extension half
• kv-cache · gqa-mqa · mla — the affordability floor
• attention-mechanisms