Hybrid Attention + SSM Architectures

Definition

A hybrid attention + SSM architecture interleaves a few full-attention layers with many constant-state layers — Mamba/SSM, linear-attention, or delta-rule layers — so that only the handful of attention layers pay quadratic cost and hold a full-context KV cache. By 2026 the frontier norm is roughly one attention layer per 3–8 constant-state layers, giving near-Transformer quality with a fraction of the cache and much higher throughput. Two structural styles exist: sequential (whole layers are one type or the other) and parallel (both mechanisms coexist as heads inside one layer).

Key math / How it works

The leverage is the state-space recurrence: an SSM/linear layer carries a fixed-size state $h_t=A{h}_{t-1}+B{x}_t$ updated in $O(1)$ per token, so its aggregate cache is constant in sequence length — unlike attention’s $O(n)$-growing KV cache. Sprinkling a small fraction of true attention layers back in restores the exact global recall that pure SSMs lack.

Sequential hybrids (all [arxiv-verified]):

• Nemotron-H (NVIDIA, arXiv:2504.03624) replaces ~92% of self-attention with Mamba-2; attention is only ~8% of layers, evenly dispersed. The 8B stack is 4 attn / 24 Mamba / 24 FFN (of 52 layers); the 56B is 10 / 54 / 54 (of 118). Uses GQA-8 and NoPE, and reaches ~2.4× throughput versus a same-size Transformer, with a tiny aggregate KV cache because the Mamba layers are constant-memory.
• Nemotron Nano 2 (arXiv:2508.14444) — a newer hybrid reasoning model that fits 128k context on a single 22 GB GPU.
• Jamba (arXiv:2403.19887) — the early blocks-of-(Transformer + Mamba) + MoE recipe.

Parallel hybrids ([arxiv-verified]):

• Hymba (NVIDIA, arXiv:2411.13676) runs attention heads and SSM heads within the SAME layer, fused. It adds meta tokens, cross-layer KV sharing, and a sliding window, achieving 11.67× cache reduction and 3.49× throughput at 1.5B scale.

Delta-rule hybrids already covered elsewhere fit the same pattern: Kimi K3 (3 KDA : 1 MLA, see kda), Qwen3.8-Next (3 GDN : 1 attention, see gated-deltanet).

Why it matters

Attention’s KV cache is the binding constraint on long-context serving; constant-state layers slash it while a thin residue of attention layers preserves precise recall. The empirically converged ratio — about one attention layer per 3–8 constant-state layers — is now standard across NVIDIA (Nemotron-H/Nano/Hymba), Moonshot (Kimi K3), Alibaba (Qwen3.8-Next), and AI21 (Jamba), making hybrids the default answer to “how do we get Transformer quality without Transformer memory.”

Taught in

• 07-linear-and-subquadratic — SSM/linear/delta layers and why hybrids beat pure variants.
• 10-sota-synthesis — the hybrid ratio as the 2026 frontier norm.

See also

• mamba — the Mamba-2 layers used in the sequential hybrids
• state-space-models — the constant-state recurrence
• kda — Kimi delta-rule hybrid (3 KDA : 1 MLA)
• gated-deltanet — Qwen3.8-Next hybrid (3 GDN : 1 attn)
• kv-cache — the cost hybrids exist to cut