Learn: Positional Information (Sinusoidal → RoPE → ALiBi)

What you're learning

Self-attention has no idea what order its tokens are in — you must inject that. This lesson walks the four canonical ways to do it (sinusoidal, learned, RoPE, ALiBi), building from the intuition of why order matters, to the math of each scheme, to why RoPE became the modern default and how it (and ALiBi) unlock long-context extrapolation. By the end you should be able to derive the RoPE relative-position identity $q_m^{\top }{R}_{m-n}{k}_n$ from scratch and explain the absolute-vs-relative distinction mechanistically.

This lesson assumes 01-scaled-dot-product-attention and 02-multi-head-attention-and-the-block — you should already be comfortable with $S=Q{K}^{\top }/\sqrt{d}$, the softmax, and the multi-head block. Here we ask the question those lessons deferred: where does position come from?

---

1. Learning map

graph TD
    A["Self-attention is<br/>permutation-equivariant<br/>(no order info)"] --> B["Must inject position"]
    B --> C["Absolute<br/>(tag each position)"]
    B --> D["Relative<br/>(what matters is offset m-n)"]
    C --> C1["Sinusoidal<br/>(Vaswani 2017)"]
    C --> C2["Learned table<br/>(GPT/BERT)"]
    C2 --> C3["⚠ can't extrapolate<br/>past L_max"]
    C1 --> D
    D --> E["RoPE<br/>rotate Q,K by position<br/>(Su 2021)"]
    D --> F["ALiBi<br/>linear distance penalty<br/>(Press 2021)"]
    E --> G["Modern default<br/>+ context extension"]
    F --> H["Strong length<br/>extrapolation"]
    G --> I["→ long-context race<br/>(lesson 09)"]
    H --> I

    style A fill:#a44,color:#fff
    style E fill:#44a,color:#fff
    style F fill:#483,color:#fff
    style I fill:#a83,color:#fff

---

2. Intuition: attention can’t see order

Recall the core of lesson 01: for a single head, the output of position $i$ is $o_i={\sum }_j{\text{softmax}}_j(q_i^{\top }{k}_j/\sqrt{d})v_j$. Look closely — nowhere does the index $j$ appear, only the content vectors $q,k,v$. The sum is over a set of key/value pairs; sets have no order.

Formally, self-attention is permutation-equivariant: permute the input tokens by any $\pi$, and the outputs come out permuted the same way, with identical values. There is no way for the mechanism to distinguish

“dog bites man” from “man bites dog”

because both contain the same bag of token embeddings $\{$dog, bites, man$\}$ — only the positions differ, and attention is blind to positions.¹ That is fatal for language: meaning lives in order. So we must inject positional information into the representations before (or during) attention.

Two philosophies for doing so:

• Absolute — give each position $m$ its own signature (a vector, or a learned tag) and stamp it onto the token there. The model learns “position 5-ness,” “position 6-ness,” etc.
• Relative — the linguistically meaningful quantity is usually the offset $m-n$ between a query at $m$ and a key at $n$ (“the adjective two words back”), not the absolute slot. A relative scheme makes the attention score depend on $m-n$ directly.

The distinction that organizes everything below

Absolute schemes answer “where am I?”; relative schemes answer “how far apart are we?”. Sinusoidal and learned encodings are absolute (though sinusoidal has a latent relative structure). RoPE and ALiBi are relative. The field moved decisively toward relative because it generalizes to lengths and offsets never seen in training — the crux of long context.

Quiz: Why does adding a positional signal before the attention matmul break permutation-equivariance, and what property of the raw attention sum made the model order-blind in the first place?

Answer

What made it blind: the output $o_i={\sum }_j{\alpha }_{ij}{v}_j$ with ${\alpha }_{ij}={\text{softmax}}_j(q_i^{\top }{k}_j/\sqrt{d})$ depends only on the content of the $q,k,v$ vectors, never on the indices $i,j$. Permuting the tokens permutes the terms of the sum but leaves each term’s value unchanged — a sum is invariant to the order of its addends — so the operation is equivariant to any permutation. Why injection fixes it: once you add a position-dependent vector to token $m$ (or rotate/bias by $m$), two tokens with identical content at different positions now carry different vectors, so $q^{\top }k$ (and hence $\alpha$) changes when you reorder them. The set becomes effectively a sequence: each element’s representation now encodes where it sits.

---

3. Math

Setup: token embeddings $x_1,\dots ,x_L\in {\mathbb{R}}^d$; queries/keys $q_m,k_n\in {\mathbb{R}}^d$ (head dim $d$; ignore multi-head indexing).

3.1 Sinusoidal absolute (Vaswani et al., 2017)

The original Transformer adds a fixed (non-learned) positional vector ${\mathrm{PE}}_m$ to each token embedding: $x_m\leftarrow x_m+{\mathrm{PE}}_m$. The signature is a bank of sinusoids at geometrically-spaced frequencies. For position $m$ and dimension pair index $i\in \{0,\dots ,d/2-1\}$:¹

$${\mathrm{PE}}_{m,2i}=\sin \left(m{\theta }_i\right),{\mathrm{PE}}_{m,2i+1}=\cos \left(m{\theta }_i\right),{\theta }_i=10000^{-2i/d}.$$

Frequencies sweep from ${\theta }_0=1$ (fast, wavelength $2\pi$) down to $\approx 10000^{-1}$ (slow, wavelength $\sim 2\pi \cdot 10^4$). Think of it as a binary-clock-in-continuous-form: low dims tick fast, high dims tick slowly, so the full vector uniquely identifies position over a huge range.

Why sinusoids specifically? Because a shift by a fixed offset $\Delta$ is a linear map on each 2D $(\sin ,\cos )$ pair — the rotation identity

$$\left(\begin{array}{c}\sin (m{\theta }_i+\Delta {\theta }_i)\\ \cos (m{\theta }_i+\Delta {\theta }_i)\end{array}\right)=\underset{\text{fixed given }\Delta }{\underbrace{\left(\begin{array}{cc}\cos \Delta {\theta }_i& \sin \Delta {\theta }_i\\ -\sin \Delta {\theta }_i& \cos \Delta {\theta }_i\end{array}\right)}}\left(\begin{array}{c}\sin m{\theta }_i\\ \cos m{\theta }_i\end{array}\right)$$

means ${\mathrm{PE}}_{m+\Delta }$ is a linear function of ${\mathrm{PE}}_m$. So the model can, in principle, learn to attend by relative offset. This latent relative structure is exactly the seed RoPE later makes explicit and exact. It’s absolute in form, weakly relative in spirit — and being parameter-free, it can be evaluated at any $m$, so it nominally extrapolates (though in practice performance still degrades well past the training length).

3.2 Learned absolute (GPT / BERT style)

Instead of a fixed formula, learn a position embedding table $P\in {\mathbb{R}}^{L_{\max }\times d}$ and add the row for each slot: $x_m\leftarrow x_m+P_m$, with $P$ trained by backprop like any other parameter. This is what original GPT and BERT use. It’s maximally flexible — the model discovers whatever positional geometry the data wants — and costs one embedding lookup.

Learned absolute encodings cannot extrapolate past $L_{\max }$

The table has exactly $L_{\max }$ rows. At inference, a position $m\ge L_{\max }$ has no row — the encoding is literally undefined, so the model simply cannot run on longer sequences without adding untrained parameters. Even within range, rows for rarely-seen large positions are undertrained. This hard length ceiling is the central practical failure of learned absolute encodings and a primary reason the field moved to relative schemes (RoPE/ALiBi) that are defined for any offset. If you only remember one drawback of any scheme in this lesson, remember this one.

3.3 RoPE — Rotary Position Embedding (Su et al., 2021)

Idea: don’t add a position vector — rotate each query and key by an angle proportional to its position, so that when you take the dot product $q_m^{\top }{k}_n$, the two rotations combine into a single rotation by the offset $m-n$. Position enters multiplicatively, and the score comes out depending only on relative distance — while each vector is still stamped with its absolute position individually.

Work in 2D first. Split the head dimension into $d/2$ coordinate pairs; each pair gets a frequency ${\theta }_i=b^{-2i/d}$ (base $b=10000$ by default, same bank as sinusoidal). Define the per-pair rotation matrix

$$R_{{\theta }_i,m}=\left(\begin{array}{cc}\cos m{\theta }_i& -\sin m{\theta }_i\\ \sin m{\theta }_i& \cos m{\theta }_i\end{array}\right),$$

and let $R_m=\mathrm{diag}(R_{{\theta }_0,m},\dots ,R_{{\theta }_{d/2-1},m})\in {\mathbb{R}}^{d\times d}$ be the block-diagonal rotation that spins every pair of $q$ (or $k$) by its own angle $m{\theta }_i$. RoPE replaces $q_m\to R_m{q}_m$ and $k_n\to R_n{k}_n$.²

The relative-position identity. The attention logit is the dot product of the rotated vectors:

$$(R_m{q}_m{)}^{\top }(R_n{k}_n)=q_m^{\top }{R}_m^{\top }{R}_n{k}_n.$$

Now use the defining property of 2D rotations: $R_m^{\top }=R_m^{-1}=R_{-m}$, and rotations by angles add, $R_a{R}_b=R_{a+b}$. Therefore, block-by-block,

$$R_m^{\top }{R}_n=R_{-m}{R}_n=R_{n-m},$$

and the logit collapses to

$$\boxed{(R_m{q}_m{)}^{\top }(R_n{k}_n)=q_m^{\top }{R}_{m-n}{k}_n}$$

(up to the sign convention on the offset; $R_{m-n}=R_{n-m}^{\top }$). The absolute positions $m$ and $n$ vanished — only their difference $m-n$ survives. That is the whole trick: absolute rotations applied independently to $q$ and $k$ produce a score that is a function of relative offset, because the dot product multiplies $R_m^{\top }$ by $R_n$ and rotations compose by adding angles.

Why RoPE is the modern default

• Relative for free, no extra params, no extra terms. It’s applied inside $q,k$ (a fixed rotation, zero learned parameters), so it needs no bias-add and no attention-score modification — it drops into any attention/FlashAttention kernel.
• Norm-preserving. Rotations are orthogonal, so $\Vert R_m{q}_m\Vert =\Vert q_m\Vert$ — RoPE never changes vector magnitudes, only phases. Clean and numerically benign.
• Multi-frequency = multi-scale. Fast pairs resolve nearby offsets, slow pairs carry long-range position — one vector spans local and global structure.
• Context extension. Because position is a continuous angle, you can extend the usable context at (or after) train time by manipulating the frequencies — e.g. linear position interpolation (scale $m\to m/s$) or NTK-aware / YaRN base-$\theta$ scaling (raise $b$) so the fastest pairs don’t alias at unseen distances. This is why nearly every modern open LLM (LLaMA, Qwen, Mistral, DeepSeek, …) uses RoPE and ships long-context variants by retuning $\theta$. We unpack this in 09-long-context-race.

Quiz: Why does RoPE yield a relative score even though the rotation is applied using each token's absolute position? (The one to really nail.)

Answer

Because the two absolute rotations meet inside a dot product, and the dot product transposes one of them. RoPE forms $R_m{q}_m$ and $R_n{k}_n$ independently using absolute positions $m,n$. The logit is $(R_m{q}_m{)}^{\top }(R_n{k}_n)=q_m^{\top }(R_m^{\top }{R}_n)k_n$. For rotation matrices, $R_m^{\top }=R_{-m}$ and angles add, so $R_m^{\top }{R}_n=R_{-m}{R}_n=R_{n-m}$ — a rotation by the difference of the two absolute angles. The absolute parts cancel exactly the way $e^{-im\varphi }{e}^{in\varphi }=e^{i(n-m)\varphi }$ cancels: each vector carries absolute phase, but the interaction keeps only the phase difference. So each token is still individually tagged with its absolute position (useful, and it’s what lets you interpolate positions), yet every attention logit depends only on $m-n$. Absolute stamping, relative interaction.

3.4 ALiBi — Attention with Linear Biases (Press et al., 2021)

RoPE bakes relativity into $q,k$. ALiBi takes an even simpler route: leave $q,k$ (and embeddings) untouched — inject no positional vectors at all — and instead add a static, non-learned linear penalty to the attention scores that grows with distance. For a query at $i$ and key at $j$ (causal, so $j\le i$):³

$${\text{logit}}_{ij}=\frac{q_i^{\top }{k}_j}{\sqrt{d}}-m\cdot (i-j),$$

where $(i-j)\ge 0$ is the distance and $m>0$ is a fixed per-head slope. The bias is $0$ at zero distance and becomes increasingly negative the further back the key is — a soft recency prior that says “attend less to far-away tokens, linearly.” Slopes are set as a fixed geometric sequence across heads (e.g. for 8 heads, $m_h=2^{-8h/H}$: $\frac{1}{2},\frac{1}{4},\dots ,\frac{1}{256}$), so different heads look back at different length scales — some sharply local, some nearly flat/global.

Because the bias is a pure function of the offset $i-j$ (defined for any distance) with no learned or stored position representation, ALiBi is inherently relative and extrapolates strongly: the original paper trained on length 1024 and matched or beat sinusoidal/RoPE baselines when evaluated at much longer lengths (e.g. 2048+), with essentially no length ceiling.³ The trade-off vs. RoPE: the linear penalty is a strong inductive bias toward recency rather than a fully general position representation, and it’s most natural for decoder (causal) attention.

Two ways to be relative

RoPE and ALiBi both make the score depend on $m-n$, but at different points in the computation. RoPE modifies the operands ($q,k$ are rotated) before the matmul — it’s a rich, phase-based relative encoding. ALiBi modifies the scores after the matmul with a scalar additive term — it’s a cheap, monotone recency bias. RoPE preserves full directional information at each offset; ALiBi throws all offset structure into a single slope per head. RoPE dominates as the general-purpose default; ALiBi remains a clean, extrapolation-friendly choice and inspired many learned-relative-bias variants (e.g. T5’s relative bias).

---

4. Illustration: RoPE as position-dependent 2D rotation

Each 2D coordinate pair of $q$ (and $k$) is spun by an angle proportional to its position; higher-frequency pairs spin faster. The dot product then depends only on the angle between the rotated query and key — i.e. on $m-n$.

One (q,k) pair, frequency θ_i

q (unrotated) R_m q (spin m·θ_i) R_n k (spin n·θ_i) ∠ = (m−n)·θ_i

d/2 pairs = d/2 frequencies
pair 0 (θ₀=1): fast spin — local offsets







pair i (mid θ_i): medium spin





pair d/2−1 (θ→1e−4): slow spin — global position




position m increases → each pair advances by m·θ_i

Figure 1 — Left: within one 2D coordinate pair, the query is rotated by $m{\theta }_i$ and the key by $n{\theta }_i$; their dot product depends only on the angle between them, $(m-n){\theta }_i$ — the relative offset. Right: the head dimension is split into $d/2$ pairs, each with its own frequency ${\theta }_i=10000^{-2i/d}$, so fast pairs encode fine local offsets and slow pairs encode coarse global position. Absolute rotation per token, relative interaction under the dot product.

---

5. FLOPs & notes: cheap to compute, huge for long context

The headline: every positional scheme here adds negligible FLOPs, yet the choice dominates long-context behavior.

• Sinusoidal / learned absolute: one add of a length-$d$ vector per token — $O(Ld)$, a rounding error next to the $O(L^{2}d)$ attention matmuls. Learned costs one extra embedding lookup.
• RoPE: rotating $q$ and $k$ is, per 2D pair, 4 multiplies + 2 adds (or via the complex/half-split trick, a handful of elementwise ops). Total $O(Ld)$ elementwise work on $Q,K$ only — no change to the $O(L^{2}d)$ score/output matmuls, and no extra HBM matrix. It’s typically fused into the attention kernel (FlashAttention exposes a RoPE-on-the-fly path), so its runtime cost is essentially free.
• ALiBi: adding a scalar $-m\cdot (i-j)$ to each of the $L^2$ scores is $O(L^2)$ adds — same order as the softmax’s own elementwise work, still trivial beside the matmul FLOPs, and it stores no $L^2$ bias matrix (the bias is computed from indices on the fly).

Negligible FLOPs, decisive consequences

Do not mistake “cheap to compute” for “unimportant.” The positional scheme sets the model’s length-generalization ceiling. Learned-absolute hard-caps at $L_{\max }$. Sinusoidal degrades past training length. RoPE’s behavior at unseen distances is governed by the base $\theta$ (base $10000$): the slowest pair must not complete a full turn within the context, or distant positions alias — which is exactly why long-context recipes raise the RoPE base (NTK-aware scaling) or interpolate positions (linear PI, YaRN) so the frequency bank still resolves the extended range. ALiBi sidesteps this with its parameter-free recency bias. The FLOPs are a footnote; the extrapolation semantics are the whole story.

This is the doorway to 09-long-context-race, where RoPE-base scaling, position interpolation, YaRN, and ALiBi/relative-bias variants are the main characters in the race to million-token context. It also connects back to the quadratic cost of attention itself — the reason long context is hard in the first place — which is the subject of the next lesson.

Quiz: Learned-absolute costs one lookup; RoPE costs a per-token rotation; ALiBi costs an $O(L^2)$ add. All are negligible vs. the matmuls. So on what axis do they actually differ, and which fails first when you double the inference length past training?

Answer

They differ on length generalization, not compute. Doubling inference length past training: learned-absolute fails hardest and first — positions $\ge L_{\max }$ have no embedding row at all, so it can’t even run without new parameters. Sinusoidal runs (the formula is defined for any $m$) but quality degrades beyond the trained range. RoPE runs and degrades more gracefully, and — crucially — can be made to extrapolate by scaling its base $\theta$ or interpolating positions (NTK/YaRN/PI), because position is a continuous rotation angle. ALiBi extrapolates most readily out-of-the-box thanks to its parameter-free monotone distance penalty. So the ranking on “survives length extrapolation” is roughly ALiBi ≈ RoPE(+scaling) > sinusoidal > learned-absolute — the inverse of how much each hard-codes a fixed position table.

Quiz: Sinusoidal encoding is called "absolute," yet the Transformer paper motivates it partly by a relative argument. What is that argument, and how does RoPE turn the same idea from "possible in principle" into "exact by construction"?

Answer

The sinusoidal relative argument: for any fixed offset $\Delta$, ${\mathrm{PE}}_{m+\Delta }$ is a linear function of ${\mathrm{PE}}_m$ — each 2D $(\sin ,\cos )$ pair maps under a shift by the fixed rotation $R_{\Delta {\theta }_i}$. So a linear projection in the attention could, in principle, learn to detect relative offsets. But it’s only latent: the encoding is added to the embedding and the model must learn to exploit the structure; nothing forces the score to depend solely on $m-n$. RoPE makes it exact: instead of adding sinusoids and hoping the model uses the linearity, it applies the rotation directly to $q$ and $k$. Then the dot product $q_m^{\top }{R}_m^{\top }{R}_n{k}_n=q_m^{\top }{R}_{m-n}{k}_n$ depends on $m-n$ by construction, for every query-key pair, with no learning required. RoPE is the sinusoidal relative property promoted from an emergent possibility to a structural guarantee.

---

6. Practice problems

Practice problem 1: derive the RoPE offset identity for one 2D pair

Problem: Take a single 2D coordinate pair with frequency $\theta$. Let $R_{\varphi }=\left(\begin{array}{cc}\cos \varphi & -\sin \varphi \\ \sin \varphi & \cos \varphi \end{array}\right)$. Show that $(R_{m\theta }q{)}^{\top }(R_{n\theta }k)$ depends on $m,n$ only through $m-n$, and identify the resulting expression.

Worked solution: Expand $(R_{m\theta }q{)}^{\top }(R_{n\theta }k)=q^{\top }{R}_{m\theta }^{\top }{R}_{n\theta }k$. Rotation matrices are orthogonal with $R_{\varphi }^{\top }=R_{-\varphi }$, and they compose additively, $R_a{R}_b=R_{a+b}$. Hence $R_{m\theta }^{\top }{R}_{n\theta }=R_{-m\theta }{R}_{n\theta }=R_{(n-m)\theta }$. So the product equals $q^{\top }{R}_{(n-m)\theta }k$ — a function of the offset $n-m$ only. Writing it out: with $q=(q_1,q_2),k=(k_1,k_2)$ and $\delta =(m-n)\theta$,
$q^{\top }{R}_{-\delta }k=(q_1{k}_1+q_2{k}_2)\cos \delta +(q_1{k}_2-q_2{k}_1)\sin \delta ,$
i.e. a linear combination of the dot product and the 2D cross-product, weighted by $\cos$ and $\sin$ of the relative angle $(m-n)\theta$. The absolute positions have completely cancelled. Stacking $d/2$ such pairs (each with its own ${\theta }_i$) gives full-dimensional RoPE, with the total logit ${\sum }_i{q}^{(i)\top }{R}_{(m-n){\theta }_i}{k}^{(i)}$ — a sum of relative-offset terms across frequencies.

Practice problem 2: choose a RoPE base for a target context length

Problem: You train with RoPE base $b=10000$, head dim $d=128$, on context length $L=4096$, and want to serve $L^{\prime }=32,768$ without retraining aliasing badly. (a) Which frequency pair is most at risk of aliasing at long range, and why? (b) Qualitatively, what does raising the base $b$ (NTK-aware scaling) do, and what does linear position interpolation ($m\to m/s$) do instead? (c) Which pairs does each method mainly protect?

Worked solution: (a) The slowest pair, $i=d/2-1$, with ${\theta }_{\max }\approx b^{-(d-2)/d}\approx 10000^{-0.984}\approx 1.2\times 10^{-4}$. Its wavelength is $2\pi /\theta \approx 5.2\times 10^4$ tokens — barely longer than $L^{\prime }=32768$. If any pair’s phase exceeds what was seen in training (each pair’s max trained angle is $L{\theta }_i$), the model meets rotations it never learned to read; the slow pairs, which carry coarse global position, are where extrapolation breaks. (Conversely the fastest pairs already wrap many times even within $L$, so they’re “used to” full-circle phases.) (b) Raising $b$ shrinks every ${\theta }_i=b^{-2i/d}$, i.e. slows all rotations, so a given position advances less phase — the highest-frequency pairs in particular stop completing extra turns at long range (NTK-aware scaling stretches the frequency spectrum, targeting the fast end while barely touching the slow end). (c) Linear PI rescales position itself ($m\to m/s$ with $s=L^{\prime }/L=8$), so every pair’s phase is uniformly compressed back into the trained range — it protects all pairs uniformly but blurs fine local resolution. NTK/base scaling stretches the spectrum so the fast pairs don’t over-rotate while preserving low-frequency (global) resolution better than uniform PI. In practice YaRN blends the two (interpolate low freqs, extrapolate high freqs). Full treatment in 09-long-context-race.

---

7. Primary source & going deeper

• Read the RoPE paper first (primary source): Su, Lu, Pan, Murtadha, Wen, Liu — “RoFormer: Enhanced Transformer with Rotary Position Embedding” — arXiv:2104.09864.² Work §3.2–3.4: the 2D formulation, the general block-diagonal rotation, and the relative-position derivation you reproduced above. Their complex-number view ($q$ as $q{e}^{im\theta }$) is the cleanest way to see why the phases subtract.
• Sinusoidal: Vaswani et al., “Attention Is All You Need” — arXiv:1706.03762, §3.5.¹
• ALiBi: Press, Smith, Lewis — “Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation” — arXiv:2108.12409.³

Ask the teacher

This lesson compresses a lot. If any step didn’t land — the rotation-composition algebra, why raising the RoPE base helps, how ALiBi slopes are chosen, or how RoPE is fused into a FlashAttention kernel — ask me for a follow-up. Good targets: “walk me through YaRN vs. NTK vs. linear PI,” “show RoPE in the complex-exponential formulation,” or “why does ALiBi struggle in bidirectional/encoder attention?“

---

8. What’s next

1. 04-quadratic-bottleneck-and-io-aware (next) — now that tokens know where they are, confront why attention is expensive: the $O(L^2)$ score matrix, the memory wall, and the IO-aware (FlashAttention) fix. Positional schemes ride inside that kernel (RoPE fused into $q,k$; ALiBi as a score bias), so the two lessons interlock.
2. 09-long-context-race (forward tease) — RoPE-base scaling, position interpolation, YaRN, and ALiBi/relative-bias variants as the levers in the million-token-context race.
3. 02-multi-head-attention-and-the-block (back) — reread the head/block structure knowing that RoPE is applied per head to $q,k$ after projection, before the score matmul.

---

Reference topic: attention-mechanisms | Glossary: attention-glossary | Concepts: rope, alibi, positional-encoding | Filed: 2026-09-10

---

References

Footnotes

1. [established] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin, “Attention Is All You Need” — arXiv:1706.03762 (NeurIPS 2017). arxiv.org/abs/1706.03762. §3.5 defines the sinusoidal positional encoding ${\mathrm{PE}}_{m,2i}=\sin (m10000^{-2i/d})$, ${\mathrm{PE}}_{m,2i+1}=\cos (m10000^{-2i/d})$, and motivates it via the linear-shift (relative-offset) property. Source of the permutation-equivariance framing. ↩ ↩² ↩³

2. [established] Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, Yunfeng Liu, “RoFormer: Enhanced Transformer with Rotary Position Embedding” — arXiv:2104.09864 (2021; Neurocomputing 2024). arxiv.org/abs/2104.09864. Source of the rotary embedding: block-diagonal rotation $R_m$ applied to $q,k$, the per-pair frequencies ${\theta }_i=10000^{-2i/d}$, and the relative-position identity $(R_m{q}_m{)}^{\top }(R_n{k}_n)=q_m^{\top }{R}_{m-n}{k}_n$. The primary source for this lesson. ↩ ↩²

3. [established] Ofir Press, Noah A. Smith, Mike Lewis, “Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation” — arXiv:2108.12409 (ICLR 2022). arxiv.org/abs/2108.12409. Source of the additive linear-distance bias $-m\cdot (i-j)$, the fixed geometric per-head slopes, and the train-short/test-long length-extrapolation results (train at 1024, evaluate at 2048+).

   ↩ ↩² ↩³