RoPE (Rotary Position Embedding)

Definition

Rotary Position Embedding (Su et al., 2021; arXiv:2104.09864) injects position by rotating each query and key by an angle proportional to its absolute position, so that the attention logit depends only on the relative offset $m-n$. It is the de-facto positional scheme in modern LLMs (LLaMA, Qwen, Mistral, DeepSeek) because it is parameter-free, norm-preserving, drops into any FlashAttention kernel, and extends to long context by retuning frequencies.

Key math

Split the head dim $d$ into $d/2$ coordinate pairs, each with frequency ${\theta }_i=b^{-2i/d}$ (base $b=10000$ default). Let $R_m=\mathrm{diag}(R_{{\theta }_0,m},\dots ,R_{{\theta }_{d/2-1},m})$ be the block-diagonal rotation spinning pair $i$ by $m{\theta }_i$. RoPE sets $q_m\to R_m{q}_m$, $k_n\to R_n{k}_n$. The logit collapses to a relative form:

$(R_m{q}_m{)}^{\top }(R_n{k}_n)=q_m^{\top }{R}_m^{\top }{R}_n{k}_n=q_m^{\top }{R}_{n-m}{k}_n,$

using $R_m^{\top }=R_{-m}$ and additive angle composition $R_a{R}_b=R_{a+b}$. The absolute positions cancel exactly the way $e^{-im\varphi }{e}^{in\varphi }=e^{i(n-m)\varphi }$ does — absolute stamping, relative interaction. Fast pairs (${\theta }_0=1$, wavelength $2\pi$) resolve local offsets; slow pairs ($\theta \approx 10^{-4}$) carry global position.

Context extension. Position is a continuous angle, so unseen distances are reached by manipulating $\theta$: linear position interpolation rescales $m\to m/s$; NTK-aware / YaRN raise the base $b$ so the fastest pairs don’t alias. The slowest pair (wavelength $\approx 2\pi /{\theta }_{\max }\approx 5\times 10^4$ tokens at $d=128$) is the one at risk when serving beyond training length. Cost is $O(Ld)$ elementwise on $Q,K$ only — negligible vs. the $O(L^{2}d)$ matmuls.

Why it matters

RoPE turned the sinusoidal “relative offset is learnable” property into a structural guarantee, and its continuous-angle form is the lever behind the entire long-context race (PI, NTK, YaRN). It is also the reason MLA needs decoupled RoPE: a position rotation sitting between the up-projection matrices breaks weight absorption.

Taught in

• 03-positional-information — §3.3 derivation, §5 the base-scaling / extrapolation semantics.

See also

• positional-encoding · alibi — sibling schemes
• mla — decoupled RoPE and the absorption clash
• flash-attention — RoPE fused on-the-fly into $q,k$
• attention-glossary · attention-mechanisms