FP8 Training

Definition

FP8 training uses 8-bit floating point for the dominant GEMM inputs, roughly doubling tensor-core throughput and halving bytes moved for those matmuls versus BF16 — a direct ~2× lever on the biggest line item of a multi-million-dollar run. Two formats: E4M3 (4 exponent, 3 mantissa — more precision, used for forward/most GEMMs) and E5M2 (5 exponent, 2 mantissa — more range, historically for gradients). The entire game is managing outliers: FP8’s tiny dynamic range and 3-bit mantissa mean one large value in a tensor blows up quantization error for everything sharing its scale. See mixed-precision-training for the BF16 baseline.

Key math / How it works

Standard large-scale training is BF16 mixed precision (BF16 matmuls, FP32 master weights + moments; Llama 3 405B was entirely BF16). FP8 goes further but only where numerically safe.

NVIDIA Transformer Engine (TE) makes FP8 practical on Hopper/Ada/Blackwell: an fp8_autocast context, fused FP8 modules, and per-tensor scaling-factor management. Recipes: DelayedScaling (history of per-tensor amax picks the next scale — the original Hopper recipe), per-tensor current scaling, and MXFP8 block scaling (Blackwell-only, hardware micro-scaling). TE is the FP8 backend under Megatron-Core and NeMo.

DeepSeek-V3’s recipe — the reference implementation (first open model pre-trained at scale in FP8, 671B, relative error vs BF16 < 0.25%):

• Fine-grained (tile/block) quantization — scale in small groups so an outlier pollutes only its own group: activations tile-wise $1\times 128$ (per token, per 128 channels), weights block-wise $128\times 128$.
• FP8 only where safe — the three Linear GEMMs Fprop / Dgrad / Wgrad run in FP8 (all E4M3, dropping the older E4M3/E5M2 hybrid). BF16/FP32 for embeddings, output head, MoE gating, normalization, attention. FP32 master weights + gradients; BF16 AdamW moments.
• Increased accumulation precision — H800 tensor cores accumulate FP8 products at only ~14-bit (up to ~2% relative error at $K=4096$); DeepSeek promotes partial sums to FP32 on CUDA cores every $N_C=128$ elements (4 WGMMAs), the minimal interval restoring precision without killing throughput.

FP8 is riskiest exactly where MoE lives — routing amplifies activation outliers/imbalance, so keeping gating and combine out of FP8 matters. “FP8 training” is a carefully chosen subset of ops, not everything in FP8.

Why it matters

FP8 changed frontier economics: DeepSeek-V3 trained a 671B MoE for a reported ~$5.6M, and Llama 4 moved to FP8 pre-training. It’s why sub-2× compute savings translate directly into millions of dollars and weeks of wall-clock. The frontier is moving lower still (MXFP8 on Blackwell, FP4 research), where DeepSeek’s outlier-management techniques become more important, not less. Note FP8 MFU looks lower against the ~1,979 TFLOP/s FP8 peak even when absolute throughput beats a 40% BF16 run.

Taught in

• 11-sota-large-scale-training — §6 mixed precision & FP8, DeepSeek-V3 recipe.

See also

• mixed-precision-training — the BF16 baseline FP8 extends
• megatron-lm — FP8 via Transformer Engine
• expert-parallelism — MoE, the most FP8-sensitive architecture
• gemm — the matmuls run in FP8
• mfu — the precision-dependent efficiency metric
• optimizer-state-memory — FP32 master + BF16 moments in the recipe
• gpu-systems-for-llms