ALiBi (Attention with Linear Biases)
Definition
Attention with Linear Biases (Press, Smith, Lewis, 2021; arXiv:2108.12409) injects position not by touching or the embeddings at all, but by adding a static, non-learned linear penalty to the attention scores that grows with query–key distance. It is a cheap, monotone recency prior whose signature property is strong length extrapolation: train short, test long.
Key math
For a causal query at and key at ():
where is the distance and is a fixed per-head slope. The bias is at zero distance and increasingly negative for far-back keys. Slopes form a fixed geometric sequence across heads — e.g. for heads, gives — so some heads are sharply local and others nearly global. Because the bias is a pure function of the offset with no learned or stored position representation, it is defined at any distance.
Cost & extrapolation. Adding a scalar to each of scores is adds (same order as softmax’s own elementwise work), and it stores no bias matrix — the value is computed from indices on the fly. The original paper trained at length 1024 and matched/beat sinusoidal and rotary baselines when evaluated at 2048+, with essentially no length ceiling.
Why it matters
ALiBi showed that a parameter-free monotone distance penalty extrapolates more readily out-of-the-box than absolute encodings, and it seeded the family of learned relative-bias variants (e.g. T5’s relative bias). Trade-off vs. RoPE: it throws all offset structure into a single slope per head (a strong recency inductive bias rather than a general relative representation) and is most natural for decoder/causal attention. RoPE remains the general-purpose default; ALiBi is the clean extrapolation-friendly alternative.
Taught in
- 03-positional-information — §3.4 the linear bias and per-head slopes; §5 extrapolation ranking.
See also
- positional-encoding · rope — sibling schemes
- sliding-window-attention — another recency-biased locality lever
- attention-glossary · attention-mechanisms