Attention FLOPs & KV-cache Calculator

A cost model for one transformer's attention. Change any field; numbers update live. Companion to the Attention Mechanisms curriculum.

Model & workload

d_model hidden size n_heads query heads n_kv_heads GQA groups (=n_heads → MHA, =1 → MQA) n_layers seq_len (s) context tokens batch (b) precision MLA latent d_c for MLA KV compare

Per-token cost (this config)

Prefill attention FLOPs

–

whole model, all layers, seq=s

Per decode-step FLOPs

–

1 new token, cache length s

KV cache (this config)

–

Attention params

–

Q,K,V,O projections, all layers

KV cache: MHA vs GQA vs MQA vs MLA

Scheme	KV heads	Cache size	vs MHA

MLA (DeepSeek-V2/V3) caches a single low-rank latent of width d_c per token per layer instead of full K,V — the current frontier KV-cache reduction. See lesson 05-mqa-gqa-mla.

Formulas used

Convention: a multiply-add = 2 FLOPs. d_h = d_model / n_heads. Softmax and normalization terms are the small non-matmul additions; MLP FLOPs are excluded (this tool isolates attention). Prefill = process all s tokens at once; decode = one new token attending to s cached keys.