Gunjan Dhanuka — Learning Notes

Learn: SOTA Attention Synthesis (Sept 2026)

Properties1
tagsattention, sota, synthesis, frontier, rope, gqa, mla, flashattention, nsa, sparse-attention, ssm, hybrids, multimodal, capstone, transformers

23 min read

Learn: SOTA Attention Synthesis (Sept 2026)

What you're learning

This is the capstone. Not new mechanics — a map of the frontier that ties every prior lesson into one working picture: what a production attention stack actually looks like in 2026, where the live research frontiers are (native trainable sparsity, the KV-cache endgame, FA-3/4 kernels, native-resolution multimodal fusion, the attention-vs-SSM question), and — the meta-skill — how to stay current after this course goes stale. By the end you should be able to spec an attention stack under a given serving constraint and defend each choice, and know which signals to watch.

This is Lesson 10, the finale of the Attention Mechanisms curriculum. It assumes the whole thing — it synthesizes rather than re-derives. Vocabulary follows the attention-glossary.

Read this first — the most time-sensitive lesson in the course

Every other lesson rests on results that are settled (softmax attention, RoPE, GQA, FlashAttention’s IO argument, MLA’s algebra). This one deliberately walks onto the moving frontier. Web search was disabled when it was written, so every 2025–2026 / current-frontier claim carries a [verify] tag and specific model configurations, arXiv IDs, and “who uses what” statements must be confirmed against primary sources before you rely on them. Treat the established landmarks (timeline backbone, MLA, FlashAttention, GQA, RoPE/YaRN) as solid; treat everything tagged [verify] as a strong prior awaiting a verification pass. The ## References are split accordingly ([established] confident, [recent]/[verify] flagged).

1. Learning map

graph TD
    A["The whole curriculum:<br/>SDPA → MHA → RoPE → O(s²) wall →<br/>KV cache → sparse → linear/SSM → VLM → long-ctx"] --> B["What frontier LLMs<br/>actually run (2026)"]
    B --> C["Positions: RoPE + YaRN-class scaling<br/>(L03, L09)"]
    B --> D["KV cache: GQA (default)<br/>or MLA (frontier) — L05"]
    B --> E["Kernels: FlashAttention-class<br/>FA-2 / FA-3 / FA-4 — L04, GPU L06"]
    B --> F["MoE FFN + hybrid<br/>attention+SSM/linear layers — L07"]
    B --> G["Native trainable sparsity:<br/>NSA / MoBA — L06 evolved"]
    C --> H["The stack you'd actually spec<br/>under a serving constraint"]
    D --> H
    E --> H
    F --> H
    G --> H
    H --> I["Open problems:<br/>subquadratic WITH full recall,<br/>KV-cache endgame, long-ctx reasoning,<br/>multimodal fusion, attention vs SSM"]
    I --> J["How to stay current<br/>(venues, groups, signals)"]

    style A fill:#44a,color:#fff
    style H fill:#4a4,color:#fff
    style I fill:#a44,color:#fff
    style J fill:#a83,color:#fff

2. The landmark timeline

Nine years, one operation, and a small number of genuine inflection points. Each line below is a load-bearing idea, linked to the lesson that derives it. Everything else in the literature is a variation, combination, or scaling-up of these.

timeline
    title Attention: landmarks 2014 → 2026
    2014 : Additive attention (Bahdanau) — attention is born as an RNN add-on (L01)
    2017 : "Attention Is All You Need" — SDPA + multi-head, attention becomes the whole model (L01, L02)
    2019 : MQA (Shazeer) — one KV head; the KV cache becomes a first-class concern (L05)
    2020 : Linear attention "Transformers are RNNs" + Performers/Linformer — dissolve the s×s matrix (L07)
    2021 : RoPE (RoFormer) — rotary positions, now the default (L03) : S4 — structured SSMs make recurrence competitive on long range (L07)
    2022 : FlashAttention — IO-aware exact attention; the memory wall, not FLOPs, is the enemy (L04, GPU L06)
    2023 : GQA — the production KV sweet spot (L05) : Mistral sliding-window + StreamingLLM sinks (L06) : Mamba — selective SSM recovers content routing (L07) : YaRN / Position Interpolation — RoPE context extension (L09) : Ring Attention — context ∝ device count (L09)
    2024 : FlashAttention-3 — Hopper asynchrony + FP8 (GPU L06) : MLA (DeepSeek-V2/V3) — low-rank latent KV, the cache frontier (L05) : Jamba / Griffin — attention+SSM hybrids go mainstream (L07)
    2025 : NSA (DeepSeek) + MoBA (Moonshot) — natively-trainable, hardware-aligned sparse attention [verify] : Linear-attention hybrids at scale (MiniMax-01 etc.) [verify]
    2026 : FA-4 / Blackwell-era kernels [verify] : hybrid attention+SSM as a default efficiency lever [verify]

The three axes every landmark moves along

Read the timeline as motion on three orthogonal axes, all introduced earlier in the course: (1) compute — kill the FLOPs (sparse L06, linear/SSM L07) or make the exact op IO-optimal (FlashAttention L04); (2) memory — shrink the KV cache (MQA→GQA→MLA L05) or replace it with constant state (SSM L07); (3) positions/length — keep attention coherent far past the training window (RoPE→YaRN, Ring Attention L09). The frontier stack is just a coherent choice on each axis at once.

3. What frontier LLMs actually use now

The remarkable thing in 2026 is how convergent the base recipe is, and how much the differentiation has moved into KV-cache strategy, sparsity, and hybridization. The common denominator:

The consensus dense-attention stack (text LLMs):

  1. RoPE for positions, almost universally, with a YaRN-class per-frequency scaling recipe for context extension (03-positional-information, 09-long-context-race). ALiBi and learned-absolute have largely lost. [verify] which exact scaling each model ships.
  2. GQA as the default KV-cache reduction (), or MLA at the frontier where the team can afford the architectural complexity (05-mqa-gqa-mla).
  3. FlashAttention-class kernels (FA-2 in most stacks, FA-3 on Hopper, FA-4 [verify] on Blackwell) — nobody materializes the matrix (04-quadratic-bottleneck-and-io-aware, 06-memory-wall-and-flashattention).
  4. MoE feed-forward almost everywhere at the top end — sparsity in the FFN (activate a few experts per token) is now as standard as sparsity in attention is becoming. (Attention lessons don’t cover MoE; it’s the other half of the frontier efficiency story — see 11-sota-large-scale-training.)
  5. A growing minority go hybrid — interleave a few full-attention layers with many SSM / linear / gated-recurrence layers (07-linear-and-subquadratic) to cut KV cost and get constant-state long-context, keeping just enough attention for exact recall.

Per-family sketch (confirm every specific against the model’s tech report):

FamilyPositionsKV strategySparsity / hybridNotes
DeepSeek V3.2 → V4RoPE (decoupled)MLA + KV compressionDSA (lightning indexer + token top-, V3.2) → CSA/HCA (V4, arXiv:2606.19348); NSA [verify]MLA+DSA+MoE is the reference efficient-frontier design; V4 ≈10% of V3.2 KV at 1M (deepseek-sparse-attention)
Llama 3 / 4RoPE + scalingGQA ()Llama-4 MoE + long-ctx [verify]The GQA-dense reference point
Qwen 2.5/3 → 3.8-NextRoPEGQA; 3.8-Next uses GDN hybrid3.8-Next: 3 GDN : 1 full-attn, + QSA trainable sparse (sourced)2026-09-12-qwen3.8-next; QSA follows DeepSeek DSA
Kimi (Moonshot) K2 → K3RoPE (K2); NoPE (K3)K3 = 3 KDA : 1 Gated-MLA hybrid (sourced)K3: constant-state KDA (channel-wise delta rule); MoBA block sparsity (arXiv:2502.13189)K3 = 2.78T MoE, 1M ctx; pairs with MoonViT (moonvit); 2026-09-12-kimi-k3
Gemma 2 / 3 / 4RoPE; per-layer base (local 10k / global 1M); pp-RoPE on global (G4)GQA + interleaved local:global (1:1 → 5:1), window 4096→1024local sliding-window + periodic global; G4 “values=keys” → −37.5% global KVsimplest widely-shipped KV control (local-global-attention); arXiv:2408.00118 / 2503.19786 / 2607.02770
GLM (Z.ai) 4.5/4.6 → 5RoPE (partial); QK-Norm4.5: GQA (96h/8kv); 5: MLAGLM-5 adopts DeepSeek DSA (tag GLM_MOE_DSA)4.5 = arXiv:2508.06471; 5 = 744B/40B MoE (2602.15763); cross-lab DSA
NVIDIA Nemotron-HNoPEGQA-8 (only ~8% of layers)Mamba-2 ↔ attention hybrid: attention ≈8% of layers (arXiv:2504.03624); Hymba parallel heads (2411.13676)most layers are constant-state Mamba → tiny KV; ~2.4× throughput (hybrid-attention-ssm)
MiniMax-01 / linear hybridsRoPEreducedlinear (“lightning”) attention hybrid [verify]large-scale bet on linear-attention layers
Jamba (AI21)—tiny (SSM)Mamba+Transformer+MoE hybridattention only in a fraction of layers (07-linear-and-subquadratic)
Gemini / GPT / Claudeundisclosedundisclosedrumored sparse/hybrid [verify]closed; infer from behavior + serving economics, not papers

Closed frontier models are inference, not fact

For Gemini / GPT / Claude the attention internals are unpublished. Anything you read about their KV strategy or sparsity is inference from latency, pricing, context limits, and occasional hints — treat it as [verify]-forever. The open frontier (DeepSeek, Qwen, Llama, Kimi, Mistral, MiniMax) is where you can actually read the architecture, which is why it anchors this lesson.

4. Native Sparse Attention & hardware-aligned sparsity

Lesson 06 taught fixed, hand-designed sparsity: sliding windows, global tokens, dilation, attention sinks. Those work, but the pattern is chosen a priori and often bolted on after pretraining, which leaves quality on the table and doesn’t co-design with the GPU. The 2025–2026 move is natively-trainable, hardware-aligned dynamic sparsity: the model learns which blocks to attend to, and the sparsity pattern is chosen to map onto efficient GPU memory access (contiguous blocks, not scattered tokens).

DeepSeek’s Native Sparse Attention (NSA) [verify] is the flagship. The idea combines three complementary branches, whose outputs are gated and summed per query:

  1. Compression (coarse global view). Compress blocks of past keys/values into a few summary tokens, so every query cheaply sees a low-resolution version of the entire context.
  2. Selection (fine-grained recall). Learnably score blocks and attend at full resolution only to the top- selected blocks — this is the trainable, content-dependent core that recovers exact-recall on the tokens that matter.
  3. Sliding window (local detail). A standard local window for recent-token precision (L06).

Two properties are the whole point, and they’re in the name:

  • Natively trainable — sparsity is present during pretraining, with a differentiable/gated block-selection path, so the model learns to be sparse rather than being pruned afterward. This avoids the quality cliff of post-hoc sparsification.
  • Hardware-aligned — the block granularity and access pattern are chosen so the kernel achieves high arithmetic intensity on GPUs (coalesced block reads, good occupancy), delivering real wall-clock speedups on long context, not just a lower FLOP count. This is the same “IO-awareness beats FLOP-counting” lesson as FlashAttention (L04), applied to sparsity.

Moonshot’s MoBA (Mixture of Block Attention) [verify] is the parallel line of work: partition the context into blocks and route each query to a small set of blocks via a top- gating mechanism — explicitly an MoE-for-attention framing (experts = key/value blocks). Like NSA it’s trainable and block-structured; the framings differ but the thesis is shared.

DeepSeek Sparse Attention (DSA) and Qwen Sparse Attention (QSA) are the sourced, shipped instances of this idea (both ingested/described from primary reports):

  • DSA (DeepSeek, referenced via the Qwen report) uses a lightweight token-level indexer to pick a top- subset of context per query, layered on MLA. Its indexer is still (token-level), so the overhead stays non-negligible at long context. Details: deepseek-sparse-attention (primary DSA paper not yet ingested — internals [verify]).
  • QSA (Qwen3.8-Next, arXiv:2608.30320 — sourced) fixes exactly that: it compresses the key sequence into micro-blocks (average-pool every tokens) before scoring, dropping the indexer to . A tiny MQA indexer (4 query heads + 1 key head, partial RoPE on 64/128 dims) scores blocks and selects the top-; config , . It’s natively trainable via two-stage continued-pretraining at 256K (dense distillation of the indexer, then joint sparse training). Payoff at 1M context vs dense GQA: ~7.6× prefill, ~4.9× decode, and RULER 512K–1M 93.0 vs 90.1 — sparse attention that gains accuracy as context grows.1 Details: qwen-sparse-attention.

The unifying thesis of 2025-26 sparsity

Move sparsity from a fixed inference-time mask to a learned, trainable, block-structured routing decision that respects the memory hierarchy. NSA and MoBA are the same bet as MoE (route each token to a few experts) applied to attention (route each query to a few KV blocks) — and the same bet as FlashAttention (the win is IO/hardware alignment, not asymptotic FLOPs). It’s the reconciliation of lessons 04, 06, and 07: sparse and trainable and hardware-aware.

Sourced vs still-to-verify

QSA (and its numbers) are transcribed from the Qwen3.8-Next primary report — sourced. NSA / MoBA / DSA internals (arXiv IDs, NSA’s exact three-branch design, top- mechanics, which shipped models use each) are still from parametric memory and carry [verify]. The direction — trainable, block-structured, hardware-aligned sparsity — is a confident synthesis; drop the DSA/NSA/MoBA [verify] tags once their primary papers are ingested.

5. The KV-cache frontier — MLA is the current best

The KV cache is the inference bottleneck (L05, 12-llm-inference-systems), and the frontier answer as of 2026 is Multi-head Latent Attention (MLA). The full derivation — low-rank latent compression, weight absorption (fold the up-projections into / so you never materialize per-head K/V), and the decoupled RoPE key that’s mandatory because RoPE’s position rotation doesn’t commute with the absorbed matrices — is in 05-mqa-gqa-mla. The one-line summary for the synthesis: MLA dominates GQA on both memory and quality simultaneously (cache equivalent to GQA-2.25 at DeepSeek-V2 dims, at MHA-level quality) — it’s a strict improvement on GQA’s tradeoff curve, not a point on it.

The broader KV frontier stacks multiple orthogonal levers, all of which cross-link to the serving-systems view:

  • Architectural width reduction — MLA (latent), or GQA (fewer heads). The biggest lever, chosen at pretraining.
  • Precision reduction — KV quantization (int8/int4/FP8 caches); orthogonal to the head/latent choice (12-llm-inference-systems).
  • Length/pattern reduction — sliding-window + attention sinks (L06) for a bounded cache, or NSA/MoBA compression branches (§4) that store compressed summaries instead of every token.
  • Elimination — hybrid delta-rule/SSM/linear layers (L07) replace the growing cache with constant-size state in most layers; only the few full-attention layers keep a real cache. The sourced 2026 exemplar is Kimi K3’s KDA (channel-wise-gated delta rule): a 3 KDA : 1 Gated-MLA hybrid where 69 of 93 layers hold only fixed-size states, making 1M context affordable (kda, 07-linear-and-subquadratic).2 Qwen3.8-Next does the same with Gated DeltaNet.1
  • Systems — PagedAttention, prefix/KV reuse, offloading, disaggregated prefill/decode — the serving layer that makes any of the above deployable (12-llm-inference-systems).

The KV cache is where architecture meets economics

Every serving cost conversation eventually reduces to bytes per token per layer in the KV cache times concurrency times context. That’s why MLA — which attacks the width of what’s cached — was the most consequential attention architecture change since GQA. When you read a new frontier model’s report, the KV-cache section tells you more about its serving economics than its parameter count does.

6. The kernels frontier — FlashAttention-3/4

The kernel side is covered mechanistically in 04-quadratic-bottleneck-and-io-aware and 06-memory-wall-and-flashattention — don’t re-derive it here. The synthesis view:

  • FlashAttention (1/2) established the thesis: exact attention, IO-aware online-softmax tiling, never materialize the matrix → HBM traffic at the same FLOPs. This is the precondition for all long-context work (L09).
  • FlashAttention-3 [verify] re-optimizes for Hopper (H100) by exploiting the architecture’s asynchrony: overlap the WGMMA tensor-core matmuls with softmax and with TMA async memory copies (warp-specialized producer/consumer pipelines), and support FP8 attention. Net: much higher H100 utilization than FA-2, which was tuned for Ampere.
  • FlashAttention-4 [verify] — the Blackwell-era successor. Expect the same pattern: co-design with the newest tensor-core/async features (Blackwell’s 5th-gen tensor cores, new low-precision formats, larger async pipelines). Confirm existence, version, and exact feature set against primary sources — this is a fast-moving, heavily [verify] claim.

The kernel lesson generalizes

The FlashAttention arc (1→2→3→4) is the same story each time: a new GPU generation exposes new async / low-precision primitives, and the attention kernel is re-written to co-design with them. The op is unchanged; the implementation chases the hardware. This is why “which FA version” is a hardware question (Ampere→Hopper→Blackwell) more than an algorithms question — and why NSA/MoBA (§4) explicitly frame themselves as hardware-aligned: they learned from FlashAttention that the kernel-hardware fit is where the real speedup lives.

7. The multimodal frontier [verify]

Lesson 08 established the two fusion families — cross-attention (Flamingo/BLIP-2, LM reaches into vision) vs. project-and-concatenate (LLaVA, vision becomes tokens in the stream) — and the punchline that visual token count drives the bill. Lesson moonvit covers native-resolution encoding. The 2026 frontier:

  • Native-resolution / any-resolution vision is now the default at the top end. Encode the image at its true aspect ratio and size in one pass (NaViT packing, MoonViT) rather than squashing to — see moonvit, moonvit, navit-packing, global-native-encoding. The cost: a single high-res image is thousands of patch tokens, so aggressive token compression (pixel-shuffle, resamplers) before the LM is mandatory.
  • Concatenate-style fusion has largely won at the frontier [verify] — Qwen-VL / InternVL / Kimi-VL / LLaVA-lineage models mostly project visual tokens into the LM stream and use ordinary self-attention, because it’s simple, scales to interleaved image/video/text, and rides all the text-side attention machinery (RoPE, GQA/MLA, FlashAttention) for free. Cross-attention (Flamingo-style) persists where you must keep a frozen LM or cap visual tokens hard.
  • Video and many-image push token counts to the point where the same efficiency toolkit from the text side reappears: sparse/windowed attention over spatiotemporal tokens, token merging/compression, and multimodal RoPE variants (e.g. position schemes that encode 2D/3D/temporal structure). This is where the attention curriculum’s L06/L07/L09 tools get reused verbatim on vision tokens.
  • Unified any-to-any models (shared attention over interleaved text/image/audio tokens) are the trajectory [verify]; the attention op is unchanged — the design work is tokenization, position encoding across modalities, and the token-count/compute budget.

VLM specifics move even faster than text

Which fusion strategy, resolution scheme, token-compression ratio, and multimodal-RoPE variant each current VLM uses is [verify] per tech report — the vision side of the frontier turns over faster than the text side. The stable takeaways: native resolution + heavy token compression, concatenate-fusion dominance, and “visual token count is the cost driver.”

Open problems

The honest frontier — what nobody has cleanly solved as of September 2026. These are the questions worth a research career, and each is the unfinished end of a lesson.

  1. Sub-quadratic with full recall. Linear attention and SSMs (L07) buy time and constant state, but pay a recall tax — a fixed-size state is a lossy compression, so they trail full attention on exact copying / associative recall / needle retrieval. Hybrids paper over this by keeping a few attention layers, but a pure sub-quadratic architecture that matches full-attention recall at scale remains open. Is the recall gap fundamental (information-theoretic, for any bounded state) or just unsolved? [contested]
  2. The KV-cache endgame. MLA (L05) shrank the cache dramatically; NSA/MoBA compression (§4) and hybrids (L07) shrink it further. But long-context serving is still cache-bound. How small can the cache get before quality breaks — and can it be made truly constant while keeping attention’s recall? The tension between “lossless growing cache” and “lossy constant state” is the deepest unresolved tradeoff in the course.
  3. Long-context reasoning, not just retrieval. RULER (L09) showed usable context ≪ advertised context, and that the gap is worst on aggregation and multi-hop reasoning over the whole window, not single-fact retrieval. Getting a model to reason over 1M tokens (combine dozens of facts spread across the context) rather than look up one is largely unsolved — and current sparse/compressed schemes may make it harder by discarding the tokens reasoning needs. [recent]
  4. Efficient multimodal fusion. Native-resolution vision (§6, moonvit) makes visual token counts explode; video makes it worse. The right way to compress visual information without discarding what a task needs, and to fuse modalities without paying full on the joint stream, is open — token compression is currently a blunt instrument.
  5. Attention vs. SSM — and the synthesis. Is the future attention with better sparsity/caching, SSM/linear with better recall, or hybrids that are already eating both? (L07) The 2025-26 evidence leans hybrid, but the optimal ratio, placement, and whether a genuinely new primitive supersedes both, are unsettled. [contested] The theoretical work unifying attention, linear attention, and SSMs (delta-rule / gated-linear-attention / test-time-regression views) [verify] may be where the answer comes from.

How to read an open problem

Each of these is a lesson’s warning callout, grown up. The recall gap (L07), advertised≠usable (L09), KV cache as the binding constraint (L05), visual-token explosion (L08) — the course flagged them as caveats; the frontier is the ongoing attempt to remove the caveats. That’s the shape of research: today’s [!warning] is tomorrow’s paper.

What to track — staying current after this lesson goes stale

This lesson has a short half-life. The durable skill is knowing where the signal is.

Venues & sources (primary first):

  • arXiv cs.CL / cs.LG, and the model tech reports themselves — DeepSeek, Qwen, Llama, Mistral, Kimi/Moonshot, MiniMax, AI21, Gemma reports are where real architecture details live (the open frontier). Read the architecture and KV-cache sections first.
  • NeurIPS / ICML / ICLR / ACL for peer-reviewed landmarks (FlashAttention, GQA, Mamba, NSA-class work land here). Best-paper lists are a high-signal filter.
  • Vendor/systems sources for kernels: NVIDIA (Blackwell, TMA, tensor-core generations), the FlashAttention repo/releases, vLLM / SGLang / TensorRT-LLM for what serving actually adopts (12-llm-inference-systems).

Groups whose output is worth reading on sight [verify]: DeepSeek (MLA, NSA), Tri Dao / Princeton & Together (FlashAttention, Mamba, hybrids), Albert Gu (SSMs), Moonshot AI (MoBA, MoonViT), Google DeepMind (Griffin/Hawk, Gemma, long-context), Meta (Llama, GQA lineage), and the linear-attention/RNN revival crowd (RWKV, RetNet, GLA, DeltaNet authors).

Signals that a claim matters (vs. noise):

  • Does it ship? An architecture used in a pretrained frontier model (MLA, GQA) outweighs one shown only on small-scale ablations.
  • Real wall-clock, not FLOPs. After FlashAttention, be skeptical of any efficiency claim without hardware-measured latency/throughput on current GPUs (the NSA “hardware-aligned” framing is the tell).
  • Measured with RULER-class evals, not perplexity or single-needle — for any long-context claim (L09).
  • Recall tested explicitly — for any sub-quadratic/sparse claim, look for copying/associative-recall/needle results, not just LM perplexity (L07).
  • Reproducible / open weights beats a closed-model marketing number every time.

Primary sources — start here, not with summaries

The backbone papers (all in ## References): Vaswani 2017 (SDPA), RoFormer (RoPE), FlashAttention 1/2/3, GQA, DeepSeek-V2/V3 (MLA), Mamba + S4, YaRN, Ring Attention, RULER, and the 2025 NSA/MoBA line [verify]. When a new model drops, read its own report’s architecture section before any secondary explainer — the whole point of this course was to make you able to.

Ask your teacher

This is the finale, but the frontier keeps moving and web search was off when this was written. Ask me to: run a verification pass on any [verify] claim once search is back; spec an attention stack for a concrete constraint you’re facing (context length, concurrency, GPU, latency SLA) and defend it; go deep on NSA/MoBA internals, the delta-rule/GLA unification of attention and SSMs, FA-3/4 kernel design, or multimodal RoPE. I can also build you a harder synthesis problem or update this lesson when the specifics firm up.

What’s next

You’ve finished the attention-mechanisms curriculum. From here:

  1. Back to the topic hub — attention-mechanisms — the map of all ten lessons and how they connect. Use it as your index.
  2. Deep-dive stubs to teach next (ask and I’ll build these out): [[concepts/native-sparse-attention]] (NSA/MoBA mechanics + the trainable-sparsity thesis), [[concepts/hybrid-attention-ssm]] (attention+SSM layer-ratio design, the recall/efficiency frontier), [[concepts/flashattention-3]] (Hopper/Blackwell kernel co-design), and [[concepts/multimodal-rope]] (position encoding across modalities).
  3. Systems cross-links — take the attention frontier into the serving/training stack: 12-llm-inference-systems (KV cache economics), 06-memory-wall-and-flashattention (kernels), 11-sota-large-scale-training (MoE + frontier training), 10-model-parallelism (context parallelism / Ring Attention).

References

Split by confidence per the course convention. [established] = settled, cite freely. [recent] = real but fast-moving. [verify] = stated from parametric memory with web search disabled — confirm ID, authors, and specifics before relying on it.

Established landmarks (confident):

Recent — real but fast-moving:

[verify] — frontier claims stated from parametric memory (search disabled):

Reference: attention-glossary | Topic: attention-mechanisms | Filed: 2026-09-11

Footnotes

  1. Qwen Team (Alibaba), “On the Design of Qwen3.8-Next Architecture” — arXiv:2608.30320 (2026). GDN (3:1) hybrid + QSA (compressed trainable sparse attention, , K=2048, r=4); RULER 512K–1M 93.0 vs 90.1, ~7.6×/4.9× prefill/decode at 1M. Transcribed from the ingested primary report; QSA follows DeepSeek DSA. See qwen-sparse-attention, gated-deltanet, 2026-09-12-qwen3.8-next. [recent, sourced] ↩ ↩2

  2. Kimi Team (Moonshot AI), “Kimi K3: Open Frontier Intelligence” — arXiv:2607.24653 (2026); KDA originates in “Kimi Linear” — arXiv:2510.26692 (2025). Channel-wise-gated delta-rule linear attention; 3 KDA : 1 Gated-MLA hybrid (69 KDA + 24 MLA / 93 layers), NoPE, constant-state, 1M context. Transcribed from the ingested primary report. See kda, 2026-09-12-kimi-k3. [recent, sourced] ↩

Graph View

Backlinks