Optimizer State Memory (16 bytes/param)

Definition

The 16 bytes/param accounting is the canonical “model states” budget for training a model in mixed precision with Adam: per parameter you must hold a bf16 working weight, a bf16 gradient, and — in fp32 — a master weight plus Adam’s two moments. This is the number that surprises people: training a model carries 8× the parameter count in extra state beyond the bf16 weights, and it is the memory that ZeRO/FSDP exist to shard.

Key math / How it works

For $\Psi$ parameters, per parameter:

Tensor	Precision	Bytes	Why
Working weight $W$	bf16	2	tensor-core matmuls want 16-bit
Gradient $\nabla$	bf16	2	produced by backward
Master weight	fp32	4	the copy the optimizer actually updates
Adam momentum $m$	fp32	4	1st moment
Adam variance $v$	fp32	4	2nd moment
Total		16

$\underset{\text{bf16 }W}{\underbrace{2}}+\underset{\text{bf16 }\nabla }{\underbrace{2}}+\underset{\text{fp32 master}+m+v}{\underbrace{4+4+4}}=16\text{B/param}$

The ZeRO paper factors this as $(2+2+K)\Psi$ with the optimizer multiplier $K=12$ for Adam. So model states $=16\Psi$ bytes — e.g. 7B → 112 GB (already over an 80 GB H100), 70B → 1.12 TB, 405B → ~6.5 TB.

Why an fp32 master copy? Adam’s update $\theta \leftarrow \theta -\alpha \hat{m}/(\sqrt{\hat{v}}+ϵ)$ can be tiny relative to $\Vert \theta \Vert$; in bf16’s 8-bit mantissa it rounds to zero (“swamping”). fp32 master + moments preserve the small updates; the bf16 weight is a rounded view refreshed each step for fast matmuls.

The punchline that trips people up: pure-fp32 Adam is also 16 B/param (4+4+4+4, no separate master). So mixed precision does not shrink model-state memory — you swap the fp32 params for a bf16 working copy plus an fp32 master. Mixed precision buys speed (16-bit tensor cores) and activation memory (bf16 activations), not optimizer memory. The 12 fp32 optimizer bytes are immovable either way.

What this is NOT: activations. Forward activations stashed for backward are a separate, batch/sequence-dependent budget (can rival params+grads+optim at long context) attacked by recomputation, FlashAttention, and sequence parallelism — not by ZeRO.

Why it matters

This single number is the entire motivation for sharded training. ZeRO/FSDP drive it to $16\Psi /N_d$ per GPU by sharding across the DP group; ZeRO-1 shards just the 12 optimizer bytes (~4×), ZeRO-3 shards all 16. Knowing which bytes replicate vs. shard tells you exactly which stage fits a given model on a given cluster.

Taught in

• 09-zero-and-fsdp — §3 deriving 16 bytes/param.
• 03-performance-modeling-roofline — mixed-precision Adam accounting.

See also

• zero — shards these bytes across the DP group
• fsdp — the PyTorch equivalent
• mixed-precision-training — why the fp32 master exists
• data-parallelism — replicates all 16 B/param
• recomputation — the orthogonal activation-memory lever
• gpu-systems-for-llms