RoPE Scaling (Position Interpolation, NTK-aware, YaRN)

Definition

RoPE scaling is the family of methods that extend a RoPE-trained model’s usable context beyond its training length by keeping inference-time rotation angles inside the trained range. A 4k-trained model produces garbage at 100k not because of FLOPs but because its slow (low-frequency) RoPE pairs hit rotation angles they never saw in training — a positional generalization failure. Two philosophies fix it: rescale position uniformly (Position Interpolation, arXiv:2306.15595), or rescale frequencies non-uniformly (NTK-aware / YaRN, arXiv:2309.00071).

Key math / How it works

RoPE spins each 2D pair of $q,k$ by $m{\theta }_i$ with ${\theta }_i=b^{-2i/d}$, $b=10000$. The wavelength of pair $i$ is ${\lambda }_i=2\pi /{\theta }_i=2\pi b^{2i/d}$ — tokens per full rotation. High-freq pairs (${\lambda }_i\ll L$) complete many rotations in training (safe to extrapolate); low-freq pairs (${\lambda }_i\gtrsim L$) never complete one (go OOD at long range).

Position Interpolation (PI). With scale $s=L^{\prime }/L\ge 1$, linearly compress every position: $m{\theta }_i\to \frac{m}{s}{\theta }_i$. Position $L^{\prime }-1$ then maps to $\approx (L-1){\theta }_i$ — the largest trained angle, so nothing is OOD. Needs only ~1000 fine-tuning steps. Flaw: it crushes all frequencies uniformly, including high-freq pairs that were fine, blurring short-range resolution (offset-1 tokens differ by an $s\times$ smaller angle).

NTK-aware scaling. Change the base instead of the position:

$b^{\prime }=b\cdot s^{d/(d-2)}.$

Since ${\theta }_0=b^0=1$ regardless of $b$, the fastest pair is untouched while the slowest is interpolated by $\approx s$ — spreading interpolation pressure across dimensions by frequency, with little or no fine-tuning.

YaRN (“NTK-by-parts”). Make the per-frequency decision explicit. With rotations-in-window $r_i=L/{\lambda }_i$ and thresholds $\alpha =1,\beta =32$, build a ramp ${\gamma }_i=\mathrm{clamp}(\frac{r_i-\alpha }{\beta -\alpha },0,1)$ and set

${\theta }_i^{\prime }={\gamma }_i{\theta }_i+(1-{\gamma }_i)\frac{{\theta }_i}{s}.$

• Many rotations ($r_i>\beta$) → ${\gamma }_i=1$ → keep ${\theta }_i$ (local resolution intact).
• Fewer than one ($r_i<\alpha$) → ${\gamma }_i=0$ → use ${\theta }_i/s$ (never OOD).

YaRN also multiplies logits by a length-dependent temperature $1/t$ with $\sqrt{1/t}\approx 0.1\ln s+1$, recovering softmax entropy as context grows. It extends further than PI with an order of magnitude less fine-tuning.

Why it matters

Positional extension is what makes a model coherent far past its training length, and it is nearly free in FLOPs — it only retunes RoPE frequencies. It is orthogonal to the systems side (flash-attention, ring-attention, kv-cache reduction via gqa-mqa/mla): YaRN can make a model coherent at 128k while it stays unaffordable without them. YaRN is the recipe behind many production long-context RoPE models.

Taught in

• 09-long-context-race — §3, deriving PI and YaRN from the wavelength argument.

See also

• rope — the base positional encoding being rescaled
• long-context — the umbrella problem (positions + systems + eval)
• ring-attention — the systems half of long context
• flash-attention · kv-cache · gqa-mqa · mla — affordability levers
• alibi · positional-encoding — alternative position schemes
• attention-mechanisms