Positional Encoding

Definition

Positional encoding is the injection of token-order information into self-attention, which is otherwise permutation-equivariant: the output $o_i={\sum }_j{\text{softmax}}_j(q_i^{\top }{k}_j/\sqrt{d})v_j$ depends only on the content of $q,k,v$, never on indices — so “dog bites man” and “man bites dog” are indistinguishable without it. Schemes divide into absolute (tag each position $m$) and relative (make the score depend on offset $m-n$); the field moved decisively to relative because it generalizes to unseen lengths.

Key math / taxonomy

Sinusoidal absolute (Vaswani et al., 2017; arXiv:1706.03762): add a fixed vector $x_m\leftarrow x_m+{\mathrm{PE}}_m$ with

${\mathrm{PE}}_{m,2i}=\sin (m{\theta }_i),{\mathrm{PE}}_{m,2i+1}=\cos (m{\theta }_i),{\theta }_i=10000^{-2i/d}.$

A shift by $\Delta$ is a linear (rotation) map on each $(\sin ,\cos )$ pair, so relative offset is learnable in principle — a latent structure RoPE later makes exact. Parameter-free, so it runs at any $m$ but degrades past training length.

Learned absolute (GPT/BERT): a trained table $P\in {\mathbb{R}}^{L_{\max }\times d}$, $x_m\leftarrow x_m+P_m$. Maximally flexible, one lookup — but hard-caps at $L_{\max }$: positions $\ge L_{\max }$ have no row, so it cannot run on longer sequences. This is the central failure that pushed the field to relative schemes.

Relative: RoPE (rotate operands $q,k$ — rich, phase-based, extendable via base/PI/YaRN) and ALiBi (add a scalar distance penalty to scores — cheap, monotone, strong extrapolation). All schemes add negligible FLOPs ($O(Ld)$ or $O(L^2)$ adds) vs. the $O(L^{2}d)$ matmuls; the choice governs the length-generalization ceiling, not compute. Rough extrapolation ranking: ALiBi ≈ RoPE(+scaling) > sinusoidal > learned-absolute.

Why it matters

The positional scheme is a footnote in FLOPs but the whole story for long context: it sets whether a model can be served at 128k+ tokens and how. RoPE-base scaling and ALiBi variants are the levers in the million-token-context race.

Taught in

• 03-positional-information — the full sinusoidal → learned → RoPE → ALiBi arc.

See also

• rope · alibi — the dominant relative schemes
• attention-glossary · attention-mechanisms