Attention Mechanisms in LLMs & VLMs — Curriculum

The evolution of attention — from the 2017 scaled-dot-product core to the state-of-the-art as of September 2026 — at the depth a frontier-lab research scientist needs. Each lesson is intuitive first, then the math, then illustrations and FLOPs/memory calculations.

Mission

Build complete, frontier-level command of attention: why each mechanism exists, the exact math, and the systems cost (FLOPs, KV-cache memory) — across both LLMs and VLMs, tracing the historical arc and landing on what’s actually used in frontier models today. Grounded in the learner’s role as an AI research scientist at a frontier lab.

How to work through this

Lessons are dependency-ordered and kept tight — one clear win each. Set the status property (unread→reading→done) as you go. Use the interactive Attention FLOPs & KV-cache calculator throughout — every lesson with a FLOPs/memory section references it. The attention-glossary is the canonical vocabulary; it’s adhered to in every lesson.

Progress tracker

TABLE WITHOUT ID
  file.link AS "Lesson",
  choice(status = "done", "✅ done",
    choice(status = "reading", "📖 reading",
      choice(status = "review", "🔁 review", "⬜ unread"))) AS "Status"
FROM "wiki/learn/attention"
SORT file.name ASC

Overall:

TABLE WITHOUT ID
  length(filter(rows.status, (s) => s = "done")) AS "✅ Done",
  length(filter(rows.status, (s) => s = "reading")) AS "📖 Reading",
  length(filter(rows.status, (s) => s = "unread")) AS "⬜ Unread",
  length(rows) AS "Total"
FROM "wiki/learn/attention"
SORT file.name ASC

Dependency map

graph TD
    subgraph "Tier 1 — Foundations"
        L1["01 · Scaled dot-product attention<br/>the core mechanism"]
        L2["02 · Multi-head attention<br/>& the transformer block"]
        L3["03 · Positional information<br/>sinusoidal → RoPE → ALiBi"]
    end
    subgraph "Tier 2 — Efficiency & the KV-cache era"
        L4["04 · The quadratic bottleneck<br/>& IO-aware attention"]
        L5["05 · MQA → GQA → MLA<br/>KV-cache reduction"]
        L6["06 · Sparse & sliding-window<br/>+ attention sinks"]
        L7["07 · Linear & sub-quadratic<br/>attention & the SSM turn"]
    end
    subgraph "Tier 3 — Multimodal & long context"
        L8["08 · Attention in ViTs & VLMs<br/>self- & cross-attention"]
        L9["09 · The long-context race<br/>RoPE scaling, ring attention"]
    end
    subgraph "Tier 4 — Frontier (Sept 2026)"
        L10["10 · SOTA synthesis<br/>what a frontier scientist must know"]
    end

    L1 --> L2 --> L3 --> L4 --> L5 --> L6 --> L7
    L3 --> L8
    L5 --> L8
    L4 --> L9
    L5 --> L9
    L7 --> L10
    L8 --> L10
    L9 --> L10

    style L1 fill:#44a,color:#fff
    style L5 fill:#4a4,color:#fff
    style L10 fill:#a44,color:#fff

Lessons

Tier 1 — Foundations ✅

• 01-scaled-dot-product-attention — the soft-lookup core; √d_k scaling; O(s²) FLOPs
• 02-multi-head-attention-and-the-block — heads, the transformer block, params/FLOPs, 6ND
• 03-positional-information — sinusoidal → learned → RoPE → ALiBi

Tier 2 — Efficiency & the KV-cache era ✅

• 04-quadratic-bottleneck-and-io-aware — why attention is memory-bound; online softmax; IO-awareness
• 05-mqa-gqa-mla — the KV-cache reduction arc; DeepSeek MLA (weight absorption, decoupled RoPE)
• 06-sparse-and-sliding-window — Mistral SWA, Longformer, BigBird; attention sinks / StreamingLLM
• 07-linear-and-subquadratic — linear attention, Performers/Linformer; the SSM turn (S4, Mamba)

Tier 3 — Multimodal & long context ✅

• 08-attention-in-vits-and-vlms — ViT self-attention; cross-attention fusion vs project-and-concatenate
• 09-long-context-race — RoPE scaling (PI, NTK/YaRN), ring attention, advertised-vs-usable context

Tier 4 — Frontier (Sept 2026) ✅

• 10-sota-synthesis — the frontier map: MLA, NSA/MoBA, FA-3/4, hybrids, open problems ⚠️ carries [verify] tags pending a web pass

Interactive tools

• Attention FLOPs & KV-cache calculator — live cost model for any attention config (MHA/MQA/GQA/MLA); used across the FLOPs sections.

Reference

• attention-glossary — canonical terms and notation used throughout.

Design principle

Every lesson opens with intuition (what problem, what’s the picture), then the math (exact formulas in LaTeX), then illustrations (Mermaid/SVG) and a FLOPs/memory section tied to the calculator. Every substantive claim is footnote-cited to a primary source. Connects to the systems view in gpu-systems-for-llms (esp. 06-memory-wall-and-flashattention, 12-llm-inference-systems) and the training view in rl-for-llms.