Learn: SOTA Large-Scale Training

What you're learning

How a frontier model is actually trained on a real cluster: how to pick a 3D/4D/5D parallelism config from memory and bandwidth constraints, how activation recomputation and sequence parallelism make it fit, how FP8 changed the economics, how comms are hidden behind compute, and what the public case studies (Llama 3 405B, DeepSeek-V3) really did. By the end you should be able to design a parallelism plan for a given model + cluster and estimate its GPU-hours and MFU — the core reasoning of a training-systems engineer.

This is the capstone of Tier 3. It assumes lessons 08-data-parallelism-and-collectives (DP, gradient all-reduce), 09-zero-and-fsdp (ZeRO/FSDP sharding), and 10-model-parallelism (TP/PP/EP/CP), and it reaches back to 06-memory-wall-and-flashattention (recomputation as a FLOPs-for-bytes trade) and 03-performance-modeling-roofline (MFU, arithmetic intensity). Everything below is synthesis: stitching those dimensions into a single working system.

---

1. Learning map

graph TD
    A["Tier 3 dims:<br/>DP · ZeRO/FSDP · TP · PP · EP · CP<br/>(lessons 08–10)"] --> B["Compose them:<br/>choose a 3D/4D/5D config"]
    B --> B1["Memory fit:<br/>params + optim + activations"]
    B --> B2["Bandwidth ladder:<br/>TP in NVLink, PP/DP over IB"]
    B --> B3["EP for MoE"]
    B1 --> C["Activation memory<br/>+ recomputation + seq-parallel"]
    C --> D["Mixed precision & FP8<br/>Transformer Engine · DeepSeek-V3 recipe"]
    B2 --> E["Comm optimization:<br/>overlap · bucketing · topology"]
    B3 --> E
    D --> F["Frameworks:<br/>Megatron-Core · DeepSpeed · TorchTitan · NeMo · nanotron"]
    E --> F
    F --> G["Real runs:<br/>Llama 3 405B · DeepSeek-V3 · Llama 4"]
    G --> H["Fault tolerance<br/>+ async checkpointing"]
    G --> I["What 'good' looks like:<br/>MFU targets & what erodes them"]
    H --> J["This IS how frontier<br/>models are trained"]
    I --> J

    style B fill:#44a,color:#fff
    style D fill:#a44,color:#fff
    style G fill:#4a4,color:#fff
    style J fill:#a83,color:#fff

---

2. Why this matters

A single frontier pre-training run is one of the largest computations humanity routinely performs. DeepSeek-V3 cost a reported 2.788M H800 GPU-hours (~$5.576Mat$2/GPU-hour);¹ Llama 3 405B consumed 30.84M H100 GPU-hours and 3.8×10²⁵ FLOPs over ~54 days on 16,384 GPUs.²³ At that scale, a 10% efficiency win is worth millions of dollars and weeks of wall-clock — and a naive parallelism choice can leave half your cluster idle (Llama 4 Behemoth reportedly ran at only ~19.7% of FP8 peak on 32K GPUs).⁴

Everything in this tier — DP, ZeRO, TP, PP, EP, CP — exists because no single dimension scales alone:

• Pure DP hits a memory wall: a 405B model in mixed precision needs ~16 bytes/param ≈ 6.5 TB just for weights + Adam states, far past a GPU’s 80 GB. You must shard the model.
• Pure TP hits a bandwidth wall: it demands an all-reduce inside every layer’s forward and backward, which is only affordable over NVLink — so TP can’t span nodes.
• Pure PP hits a bubble wall: pipeline fill/drain wastes GPUs unless you have many micro-batches.

The art is composing them so each dimension lives on the interconnect that can afford it, the model fits in memory, and the bubbles and comms hide behind compute. That composition — plus FP8 and recomputation to shrink what has to move and fit — is what this lesson is about.

The one-sentence version

Large-scale training is a constrained packing problem: fit the model in aggregate memory, map each parallelism dimension onto the interconnect fast enough for its traffic, and overlap everything you can — then FP8 and recomputation buy you back the memory and bandwidth you spent.

---

3. The interconnect hierarchy: the constraint everything bends around

Before choosing any config you must internalize the bandwidth ladder of a modern GPU cluster. Every parallelism decision is downstream of it.

Tier	Link	Bandwidth (per GPU)	Scope
Intra-node	NVLink 4 + NVSwitch (H100)	900 GB/s bidirectional	8 GPUs in a node
Inter-node	InfiniBand NDR / RoCE (400G)	~50 GB/s (400 Gb/s), 1 NIC/GPU	across nodes
Host	PCIe Gen5	~64 GB/s	GPU↔CPU

The intra-node vs inter-node gap is roughly 18× on H100.⁵⁶ This single number dictates the whole design:

graph LR
    subgraph "Node 0 (NVLink island, 900 GB/s)"
        G0[GPU0] --- G1[GPU1] --- G2[GPU2] --- G3[GPU3]
        G4[GPU4] --- G5[GPU5] --- G6[GPU6] --- G7[GPU7]
        G0 --- G7
        G3 --- G4
    end
    subgraph "Node 1 (NVLink island)"
        H0[GPU0] --- H1[GPU1]
        H2[".."] --- H7[GPU7]
    end
    G0 -. "InfiniBand ~50 GB/s" .- H0
    G1 -. IB .- H1

The placement rule that falls out of the ladder

Put the chattiest dimension on the fastest link. Tensor parallelism all-reduces twice per layer — it must stay inside the NVLink island (TP ≤ 8 on an 8-GPU node). Pipeline parallelism sends only a single activation tensor per micro-batch across a stage boundary — it tolerates InfiniBand. Data parallelism syncs gradients once per step and can be fully overlapped — it’s the most latency-tolerant, so it goes outermost. This ordering is not a heuristic; it’s forced by the 18× cliff.

DeepSeek-V3 makes the constraint vivid: it ran on H800 GPUs (an export-restricted SKU with reduced NVLink bandwidth), and its authors state they trained without tensor parallelism at all because “TP is inefficient under limited NVLink bandwidth” — instead leaning on expert parallelism, pipeline parallelism, and FP8.¹ The interconnect ladder literally reshaped the model-system co-design.

"NVLink bandwidth" is quoted inconsistently — pin down which number

Vendor “900 GB/s” for H100 NVLink is bidirectional aggregate (≈450 GB/s each way).⁵ H800’s NVLink is lower; the DeepSeek-V3 paper reasons about ~160 GB/s effective NVLink vs ~50 GB/s effective IB (a ~4:1 scale-up:scale-out ratio), which differs from the ~400 GB/s H800 spec.⁷ When you quote a bandwidth, say whether it’s spec, bidirectional, or achieved.

---

4. Composing the dimensions: how to actually choose a config

Here is the reasoning a training engineer runs, in order. The dimensions form a nested hierarchy on the GPU grid; the canonical ordering (Llama 3’s) is [TP, CP, PP, DP] from innermost (highest bandwidth need) to outermost (most latency-tolerant).³

Step 1 — Does one instance of the model fit at all? Compute the memory budget (§5). If weights + optimizer states don’t fit in one GPU, you need model sharding (TP and/or FSDP/ZeRO) before you even think about throughput.

Step 2 — Set TP to the NVLink island size, but no bigger. TP splits each matmul and all-reduces the partial sums inside every layer. Its comms are only affordable intra-node, so TP ≤ (GPUs per node) — almost always TP = 8 on H100/H800 nodes (or smaller if the model is small enough to skip it). Enable sequence parallelism alongside TP for free activation-memory savings (§5).

Step 3 — Use PP to span nodes and cut per-GPU weight memory. Pipeline parallelism partitions layers across node groups; only boundary activations cross IB. Choose PP so that (weights + optimizer)/(TP×PP) fits in HBM with room for activations. Pick enough micro-batches to keep the bubble small (bubble fraction ≈ (PP−1)/(m + PP−1) for 1F1B).

Step 4 — Add CP if sequences are long. Context parallelism shards the sequence dimension so activation memory and attention compute scale with s/CP. You only pay for it when s is large — Llama 3 kept CP=1 at 8K context and raised it to CP=16 only for the 128K long-context phase.³

Step 5 — Add EP if the model is MoE. Expert parallelism places different experts on different GPUs; tokens are routed to their experts via all-to-all. EP replaces TP as the primary intra-expert sharding for MoE and is what lets DeepSeek-V3 spread 256 experts across 64-way EP.¹

Step 6 — DP/FSDP fills the rest, outermost. Whatever GPUs remain become the data-parallel dimension: DP = total_GPUs / (TP×CP×PP×EP). Use ZeRO/FSDP to shard optimizer states (and optionally params/grads) across DP so per-GPU memory drops further. DP goes outermost because its single gradient all-reduce/reduce-scatter per step is the most tolerant of latency and the easiest to overlap.

The master constraint in one equation

$\text{total GPUs}=\underset{\text{model sharding}}{\underbrace{\text{TP}\times \text{CP}\times \text{PP}}}\times \underbrace{(\text{EP if MoE})}\times \underset{\text{replicas}}{\underbrace{\text{DP}}}$
with the placement rule TP innermost (NVLink) → CP → PP → DP outermost (IB), and the memory constraint that one shard (params+optim+activations)/(TP·PP·…) must fit in 80 GB.

Two worked configs from the case studies make this concrete:

Run	GPUs	TP	CP	PP	EP	DP	Notes
Llama 3 405B (8K ctx)	16,384	8	1	16	—	128	dense; MFU 41%3
Llama 3 405B (128K ctx)	16,384	8	16	16	—	8	CP scaled up for long context3
DeepSeek-V3	2,048	0 (none)	1	16	64	ZeRO-1	MoE; TP dropped for FP8+EP1

Quiz: You have 1,024 H100s (128 nodes × 8 GPUs) and a dense 70B model, 8K sequence length. Sketch a config and justify every dimension.

Answer

TP = 8. Keep tensor parallelism inside one node so its per-layer all-reduces ride NVLink (900 GB/s), never IB. 70B is too big to skip sharding entirely, and TP=8 immediately cuts per-GPU weight/optimizer memory 8×. Enable sequence parallelism with it.
PP: 70B in mixed precision is ~70B×16 B ≈ 1.12 TB of weights+Adam. After TP=8 that’s ~140 GB/GPU — still over 80 GB, so you need more sharding. Either PP = 2–4 (spanning nodes over IB) or rely on FSDP/ZeRO across DP to shard optimizer states. A clean choice: TP=8, PP=2, giving ~70 GB/GPU of model state, leaving room for activations + recompute.
CP = 1. 8K context is short enough that activation memory is manageable with recomputation; save CP for long-context phases.
DP = 1024/(8×2) = 64, outermost, with ZeRO-1 (shard optimizer states across the 64 DP ranks) to reclaim the Adam-state memory and overlap the gradient reduce-scatter with backward.
Sanity: TP×PP×DP = 8×2×64 = 1024 ✓. Chattiest dim (TP) on NVLink, PP across a few nodes, DP outermost and overlapped. (A pure-FSDP alternative — TP=8, PP=1, FSDP over DP=128 — is also viable and simpler; TorchTitan would express either.)

---

5. Activation memory & recomputation at scale

Weights and optimizer states are a fixed cost. Activations are the cost that explodes with batch and sequence length — and they’re what recomputation (lesson 06) and sequence parallelism attack.

5.1 The activation-memory formula

For a standard transformer layer, storing all activations for the backward pass costs, per layer, in mixed precision:⁸

$${\text{mem}}_{\text{act}}=sbh\left(34+5\frac{as}{h}\right)\text{bytes}$$

where $s$ = sequence length, $b$ = micro-batch, $h$ = hidden dim, $a$ = attention heads. The two terms have very different characters:

• The $34sbh$ term is linear in $s$ — the MLP, projections, layernorms, residuals.
• The $5\frac{as}{h}\cdot sbh=5a{s}^{2}b$ term is quadratic in $s$ — it comes from the attention score matrix ($Q{K}^{\top }$, softmax, dropout, attention-over-$V$). At long context this term dominates and is exactly the memory FlashAttention and recompute target.

5.2 Three levers, from cheapest to most aggressive

Selective activation recomputation drops only the quadratic attention term by recomputing those cheap-FLOP/heavy-memory ops in the backward pass, leaving $34sbh$ per layer.⁸ Because attention ops are low arithmetic intensity, the recompute overhead is only a few percent (~2–7%). This is the default at scale.

Full activation recomputation stores only the layer input ($2sbh$ per layer) and recomputes everything — maximal memory savings but a ~30–40% compute overhead.⁸ Used only when you’re desperate for memory.

Sequence parallelism (SP) partitions the activation-heavy but non-tensor-parallel regions (layernorm, dropout, residual) along the sequence dimension across the TP group, so the whole activation term divides by the TP degree $t$:⁸

$${\text{mem}}_{\text{act}}^{\text{SP+selective}}=\frac{34sbhL}{t}(\text{across }L\text{ layers})$$

Combined, SP + selective recompute delivered a reported ~5× activation-memory reduction and >90% cut in recomputation overhead versus full recompute in the Megatron study.⁸

graph LR
    A["Full activations<br/>sbh(34 + 5as/h)"] -->|"selective recompute<br/>(drop attn term)"| B["34·sbh<br/>~2–7% overhead"]
    A -->|"full recompute<br/>(store input only)"| C["2·sbh<br/>~30–40% overhead"]
    B -->|"+ sequence parallel ÷t"| D["34·sbh / t<br/>~5× total saving"]
    style B fill:#4a4,color:#fff
    style D fill:#44a,color:#fff

Recompute erodes MFU but not HFU — and that's the point

Recomputed FLOPs are real work the hardware does, so they count toward HFU (hardware FLOPs utilization) but not MFU (model FLOPs utilization) — MFU counts only the forward+backward FLOPs the model requires (§9).⁹ Selective recompute is chosen precisely because it buys large memory savings for a small MFU hit; full recompute’s 30–40% overhead shows up directly as an MFU gap. This is the same FLOPs-for-bytes bargain as FlashAttention’s backward pass (lesson 06), now applied layer-wide.

Quiz: A 128K-context training step is OOMing on activation memory. You can turn on (a) full recompute, (b) selective recompute + sequence parallelism, or (c) context parallelism. Which do you reach for first, and why is the quadratic term the crux?

Answer

At 128K context the $5a{s}^{2}b$ quadratic term dominates activation memory — it grows with $s^2$ while the $34sbh$ term grows with $s$. So the highest-leverage move is anything that kills or shards that term.
First reach: (b) selective recompute + SP. Selective recompute eliminates the quadratic attention term outright (recomputing it in backward at ~few-% cost), and SP divides the remaining $34sbh$ by the TP degree. Cheap and targeted.
Then (c) context parallelism if still OOM: CP shards the sequence dimension itself across GPUs, so both terms scale as $s/\text{CP}$ — this is exactly why Llama 3 raised CP to 16 only for its 128K phase.³ (a) full recompute is the last resort — it saves the most memory but its 30–40% overhead is a big MFU hit you take only if (b)+(c) still don’t fit. The general principle: match the tool to which term is exploding.

---

6. Mixed precision & FP8 training at scale

6.1 The precision ladder

Standard large-scale training is BF16 mixed precision: BF16 for the matmul inputs and forward/backward compute, an FP32 master copy of the weights and FP32 optimizer moments for the update. BF16’s 8-bit exponent (same range as FP32) is why it replaced FP16 — no loss-scaling gymnastics needed. Llama 3 405B was trained entirely in BF16.³

FP8 goes further: it uses 8-bit floats (E4M3: 4 exponent, 3 mantissa; E5M2: 5 exponent, 2 mantissa) for the GEMM inputs, roughly doubling tensor-core throughput and halving the bytes moved for those matmuls. The catch is the tiny dynamic range and 3-bit mantissa: a single outlier in an activation/weight tensor blows up the quantization error for every other value sharing that scale factor. Managing outliers is the entire game of FP8 training.

6.2 NVIDIA Transformer Engine

The Transformer Engine (TE) is NVIDIA’s library that makes FP8 training practical on Hopper/Ada/Blackwell.¹⁰ It provides an fp8_autocast context and fused FP8 transformer modules, and manages the per-tensor scaling factors that keep values in FP8’s representable range. Its recipes:¹⁰

• DelayedScaling — maintains a history of per-tensor amax to pick the next step’s scale (the original Hopper recipe).
• Per-tensor current scaling — computes the scale from the current tensor dynamically.
• MXFP8 block scaling — Blackwell-only micro-scaled FP8 with hardware block scaling.

TE is the FP8 backend under Megatron-Core and NeMo, so most FP8 training you’ll touch runs through it.¹⁰

6.3 DeepSeek-V3’s FP8 recipe — the reference implementation

DeepSeek-V3 was the first open model to successfully pre-train at scale in FP8,¹⁴ and its recipe is the canonical worked example. The key ideas:¹

Fine-grained (tile/block) quantization. Instead of one scale per tensor (which any outlier ruins), it scales in small groups so the outlier only pollutes its own group:

• Activations: tile-wise $1\times 128$ (per token, per 128 channels).
• Weights: block-wise $128\times 128$ (per $128\times 128$ block).

FP8 only where it’s safe. The three Linear-layer GEMMs — Fprop (forward), Dgrad (activation gradient), Wgrad (weight gradient) — run in FP8 (all in E4M3, dropping the older E4M3/E5M2 hybrid). Everything numerically sensitive stays higher precision: embeddings, output head, MoE gating, normalization, and attention are BF16/FP32; master weights and gradients are FP32; optimizer (AdamW) moments are BF16.¹

Increased accumulation precision. The H800 tensor cores accumulate FP8 products at only ~14-bit precision (up to ~2% relative error at K=4096). DeepSeek promotes partial sums to FP32 on the CUDA cores every $N_C=128$ elements (4 WGMMAs) — the minimal interval that restores precision without killing throughput.¹

Result: relative accuracy loss vs BF16 < 0.25% — small enough to train a 671B model to completion.¹

graph TD
    subgraph "FP8 (2× throughput)"
        F1["Fprop GEMM"]
        F2["Dgrad GEMM"]
        F3["Wgrad GEMM"]
    end
    subgraph "BF16 / FP32 (sensitive)"
        H1["Embeddings · Output head"]
        H2["Attention · Softmax"]
        H3["Norms · MoE gating"]
        H4["Master weights (FP32)<br/>Gradients (FP32)<br/>AdamW moments (BF16)"]
    end
    Q["Fine-grained quant:<br/>act 1×128 tile · weight 128×128 block<br/>E4M3 · FP32 accumulate every 128"] --> F1 & F2 & F3
    style Q fill:#a44,color:#fff

6.4 Why FP8 matters for frontier economics

FP8 roughly doubles matmul throughput and halves the memory/bandwidth for the dominant GEMMs. On a run that costs millions of dollars, that is a direct ~2× lever on the biggest line item. It’s why DeepSeek-V3 trained a 671B MoE for a reported ~$5.6M,¹ and why Llama 4 moved to FP8 pre-training.⁴ The frontier is now moving toward even lower precision (MXFP8 on Blackwell, FP4 research), where the outlier-management techniques DeepSeek pioneered become more important, not less.

FP8 is riskiest exactly where MoE lives

MoE models are especially prone to FP8 instability because routing amplifies activation outliers and imbalance.⁴ This is why the safe operations list matters: DeepSeek keeps gating and combine out of FP8. Don’t read “FP8 training” as “everything in FP8” — it’s a carefully chosen subset with FP32 accumulation and fine-grained scaling around it.

Quiz: Your MoE model trains stably in BF16 but diverges after ~2000 steps in naive per-tensor FP8. Give two mechanistic fixes and say what each addresses.

Answer

1. Fine-grained (tile/block) quantization instead of per-tensor. Divergence in FP8 almost always traces to outliers: one large activation forces a large scale factor, crushing all the small values in the same tensor to zero mantissa. Scaling in $1\times 128$ activation tiles / $128\times 128$ weight blocks confines each outlier’s damage to its own group — this is DeepSeek-V3’s core trick.¹
2. Keep the sensitive ops out of FP8 — run MoE gating, normalization, attention softmax, embeddings, and the output head in BF16/FP32, and keep FP32 master weights + FP32 accumulation (promote to CUDA cores every 128 elements). Gating in particular controls routing; small errors there cascade into load imbalance and instability. MoEs are the most FP8-sensitive architecture,⁴ so the “FP8 only for the three Linear GEMMs” discipline is what buys stability. (A third lever: current/delayed scaling via Transformer Engine to track amax over time.¹⁰)

---

7. Communication optimization

Getting the config right places the traffic on the right links; hiding the traffic behind compute is what actually recovers the FLOPs.

Overlap comms with compute. The dominant technique. In FSDP/ZeRO, the gradient reduce-scatter (and the next layer’s parameter all-gather) are launched on a separate CUDA stream so they run while the backward pass computes other layers.¹¹ In pipeline parallelism, DeepSeek-V3’s DualPipe takes this to the extreme: it schedules forward and backward micro-batches bidirectionally and splits each chunk into (attention, all-to-all dispatch, MLP, all-to-all combine) so the MoE all-to-all is fully hidden behind computation, driving the cross-node comms overhead toward zero despite EP’s ~1:1 compute-to-communication ratio.¹

Gradient bucketing. DP gradients are grouped into fixed-size buckets; a bucket’s all-reduce fires as soon as its last gradient is ready, overlapping with the still-running backward and amortizing per-message latency over larger transfers. This is standard in PyTorch DDP and the reason small gradients don’t each pay a separate kernel-launch + handshake cost.¹¹

Custom kernels when the framework isn’t enough. DeepSeek wrote custom PTX all-to-all kernels and dedicated 20 SMs as 10 communication channels to saturate NVLink+IB without stealing too much compute from the tensor cores.¹ At the frontier, comms is a kernel-engineering problem, not just a config flag.

Topology-aware placement. Rank assignment is chosen so that high-bandwidth dimensions map to physically close GPUs: TP ranks share an NVSwitch, PP/DP ranks are laid out to use the fat-tree efficiently, and InfiniBand SHARP (in-network reduction) accelerates all-reduce in the switch silicon.⁵ Llama 3’s [TP, CP, PP, DP] ordering is explicitly described as “optimized for network communication,” with DP outermost because it “can tolerate longer network latency.”³

The overlap mental model

A step’s wall-clock is $\max (\text{compute},\text{comms})$ if perfectly overlapped, but $\text{compute}+\text{comms}$ if serialized. Every technique here — buckets, separate streams, DualPipe, prefetched all-gather — is trying to push you toward the $\max$. When MFU is low and the profiler shows GPUs idle waiting on NCCL, you have an overlap failure, not a FLOPs problem.

---

8. The framework landscape (Sept 2026)

You don’t build this from scratch. Five systems dominate; know what each is for.^1213141516

Framework	Owner	What it’s for
Megatron-LM / Megatron-Core	NVIDIA	The reference for TP/PP/EP/CP + sequence parallelism. Megatron-Core is the composable library (layers, parallelism, optimizers) that others build on; Megatron-LM is the example training stack. FP8/FP4 via Transformer Engine.12
DeepSpeed	Microsoft	Home of ZeRO (stages 1/2/3), ZeRO-Offload/Infinity (CPU/NVMe offload for extreme scale), ZeRO++. The go-to for memory-constrained sharding.13
TorchTitan	PyTorch	Native-PyTorch pretraining platform: FSDP2 (per-parameter sharding), TP (incl. async-TP), PP, CP → composable 3D/4D parallelism, torch.compile, and Float8. The clean-room “how PyTorch intends you to do this” reference.14
NeMo	NVIDIA	End-to-end framework (pretrain → post-train → RL, incl. speech/multimodal) built on Megatron-Core as its backbone (via Megatron-Bridge + Transformer Engine). Now an umbrella of repos under the NVIDIA-NeMo org, shifting to PyTorch-native loops.15
nanotron	Hugging Face	Minimal, readable 3D-parallelism library (DP+TP+PP, +EP for MoE). Trained SmolLM3; paired with the educational picotron. Reach for it to understand the mechanics.16

Two cross-cutting pieces glue these together: Transformer Engine (FP8/MXFP8 kernels, §6.2) and PyTorch’s FSDP2 (fully_shard, per-parameter DTensor sharding — FSDP1 is deprecated).¹¹¹⁰

Which to pick

Want the most battle-tested TP/PP/EP for a huge dense/MoE model on NVIDIA hardware → Megatron-Core (or NeMo on top of it). Want pure-PyTorch, torch.compile-native, easy to hack, FSDP2+Float8 → TorchTitan. Memory-bound and need aggressive offload → DeepSpeed ZeRO. Want to read and learn the whole 3D stack in a weekend → nanotron/picotron.

---

9. What “good” looks like: MFU at scale

MFU (Model FLOPs Utilization), from the PaLM paper, is the ratio of observed throughput to what the hardware could do at peak FLOPs, counting only the forward+backward FLOPs the model architecturally requires (the ~$6N$-per-token accounting) — excluding recomputation.⁹ It’s implementation-independent, so it’s the fair cross-system metric.

HFU (Hardware FLOPs Utilization) counts all FLOPs actually executed, including recomputation and redundant TP work. So HFU ≥ MFU always, and the gap is your implementation overhead.⁹

Reference points (all measured, cite-able):

Run	MFU	Notes
PaLM 540B	46.2% (57.8% HFU)	the paper that defined the metric9
Megatron 530B (SP + selective recompute)	54.2%	vs 42.1% with full recompute8
Megatron 1T	56.3% MFU / 57.0% HFU	near-optimal8
Llama 3 405B (BF16)	38–43%	43% @ 8K GPUs, 41% @ 16K, 38% @ 128K ctx3
Llama 4 Behemoth (FP8)	~19.7% of FP8 peak	~390 TFLOP/s/GPU on 32K GPUs4

Rule of thumb: 35–50% MFU is good, 50%+ is excellent for large dense transformers.¹⁷ (This band is community consensus, not a single primary source.)

What erodes MFU:

• Pipeline bubbles — fill/drain idle time; grows with PP, shrinks with more micro-batches.
• Un-overlapped comms — GPUs stalling on NCCL all-reduce/all-gather/all-to-all.
• Recomputation — its FLOPs count against MFU (they’re real work not in the model’s ~6N).
• Memory-bound ops — layernorm, softmax, elementwise glue (lesson 06’s fusion targets).
• Load imbalance — especially MoE expert imbalance and uneven sequence lengths.
• Small effective batch per GPU / low arithmetic intensity — the Llama 4 lesson: 32K GPUs at only ~20% because the per-GPU work was too thin.⁴

FP8 MFU can look worse even when it's faster

When you quote “MFU” for an FP8 run, note the denominator: 19.7% of the FP8 peak (~1,979 TFLOP/s on H100) still means ~390 TFLOP/s/GPU — comparable in absolute throughput to a 40% BF16 MFU (~400 TFLOP/s vs 989 peak).⁴³ Always compare absolute TFLOP/s or fix the precision when comparing MFU numbers.

Quiz: Your run holds 45% MFU on 4K GPUs but drops to 22% when you scale the same model to 32K GPUs (same global batch). What are the two most likely culprits, and what's the fix?

Answer

Scaling GPUs at fixed global batch shrinks the per-GPU work, and two things break:
1. Comms stop hiding behind compute. More DP replicas → the same-sized gradient all-reduce/reduce-scatter is now spread over more ranks and the compute per step per GPU is smaller, so the collective no longer fits “under” the backward pass. GPUs sit idle on NCCL. Fix: raise micro-batch/global batch to restore arithmetic intensity, improve overlap (bucketing, prefetch), and use topology-aware placement / SHARP.
2. Thin per-GPU tiles + pipeline bubbles. Smaller batch per GPU means lower-arithmetic-intensity GEMMs (further from the roofline ridge) and, if you added PP, a larger bubble fraction from too few micro-batches. Fix: keep enough micro-batches per pipeline, and don’t over-shard a model that fits with less parallelism. This is essentially the Llama 4 lesson — 32K GPUs at ~20% because the work per GPU was too thin;⁴ adding context parallelism + tensor parallelism to support a larger effective batch would have helped. The general rule: MFU falls when compute per GPU drops below what’s needed to hide comms and saturate the tensor cores.

---

10. Fault tolerance & checkpointing at scale

At 10K+ GPUs, failures are the steady state, not the exception. A Meta reliability study fit an MTTF (mean time to failure) that scales inversely with GPU count: roughly ~47 days at 8 GPUs → ~1.8 hours at 16,384 GPUs.¹⁸ Llama 3 405B saw 466 job interruptions over 54 days — 419 unexpected, ~78% from hardware (GPU failures the largest single category, HBM3 second), i.e. roughly one unexpected interruption every ~3 hours; automation handled nearly all recovery and the team held >90% effective training time.³

This forces two disciplines:

Async / distributed checkpointing. You cannot stop 16K GPUs to write a multi-TB checkpoint synchronously every few hours. Solutions:

• PyTorch Distributed Checkpoint (DCP) — saves/loads sharded state in parallel across ranks (reshardable to a different parallelism layout on reload). dcp.async_save stages the state to CPU memory, then writes to storage on a background thread, so the GPU only blocks for the D2H copy.¹⁹
• GEMINI (SOSP ‘23) — checkpoints to CPU DRAM and peer-machine memory rather than remote storage, exploiting the far higher GPU↔DRAM bandwidth to cut checkpoint/recovery time by >100× and enable per-iteration checkpointing.²⁰
• CheckFreq (FAST ‘21) — pipelines a two-phase snapshot() (GPU→CPU copy overlapped with the next iteration) + persist() (async disk write), with adaptive frequency to bound overhead to a few percent.²¹

Fast, automated recovery. Detect the failure (NCCL flight-recorder / hang detection), evict the bad node, restart from the latest checkpoint on healthy hardware — with as little human intervention as possible. Llama 3 needed significant manual intervention only ~3 times across the whole run.³

Checkpoint frequency is an economics calculation

Expected work lost per failure ≈ (time since last checkpoint)/2. If MTTF is ~2 hours at your scale and a checkpoint costs seconds (async, to CPU/peer), you can checkpoint every few minutes and lose almost nothing. The whole point of GEMINI/DCP/CheckFreq is to make the checkpoint cheap enough that frequency stops being a tradeoff.

---

11. A 3D-parallel GPU layout, end to end

Putting §3–§7 into one picture — a small illustrative cluster with TP=4, PP=2, DP=2 (16 GPUs, 4 nodes):

DP replica 0
DP replica 1

Node A — PP stage 0 (layers 0..k) TP0 TP1 TP2 TP3 NVLink all-reduce every layer (900 GB/s) Node B — PP stage 1 (layers k..L) TP0 TP1 TP2 TP3 NVLink all-reduce every layer PP: activations over IB Node C — PP stage 0 (replica) TP0 TP1 TP2 TP3 Node D — PP stage 1 (replica) TP0 TP1 TP2 TP3 DP: grad reduce-scatter over IB (once/step, overlapped)

Figure 1 — Each node is a 4-GPU NVLink island holding one TP group; TP’s per-layer all-reduce never leaves the island. PP splits layers across nodes (red, activations over IB). DP replicates the whole pipeline (purple, gradient sync over IB, overlapped with backward). The chattiest dimension (TP) is innermost on the fastest link; the most latency-tolerant (DP) is outermost — exactly Llama 3’s [TP, CP, PP, DP] ordering.³

---

12. Practice problems

Practice problem 1: Design a plan and estimate GPU-hours + MFU

Problem. You must pre-train a dense 70B model on 15T tokens, sequence length 8192, on 512 H100s (64 nodes × 8 GPUs). (a) Give a parallelism plan. (b) Estimate total training FLOPs and GPU-hours at a target 40% MFU. (c) If the profiler reports 400 TFLOP/s/GPU, what MFU is that, and roughly how many wall-clock days?

Worked solution.
(a) Plan. TP=8 (NVLink island, with sequence parallelism). 70B mixed-precision state ≈ 70B×16 B ≈ 1.12 TB; after TP=8 that’s ~140 GB/GPU — over 80 GB, so add ZeRO-1/FSDP over DP to shard optimizer states (or PP=2). Take TP=8, PP=1, DP=64 with FSDP + selective recompute + SP. Check: 8×64 = 512 ✓. TP on NVLink, DP outermost and overlapped.
(b) FLOPs. Use $C\approx 6ND$ (params × tokens × 6 for fwd+bwd):
$C\approx 6\times 70\times 10^9\times 15\times 10^{12}=6.3\times 10^{24}\text{FLOPs}.$
Effective throughput per GPU at 40% MFU of H100 BF16 peak (989 TFLOP/s):
$P_{\text{eff}}=0.40\times 9.89\times 10^{14}\approx 3.96\times 10^{14}\text{FLOP/s}.$
GPU-seconds $=C/P_{\text{eff}}=6.3\times 10^{24}/3.96\times 10^{14}\approx 1.59\times 10^{10}$ s.
GPU-hours $\approx 1.59\times 10^{10}/3600\approx 4.4\times 10^6$ = ~4.4M GPU-hours. (Sanity: comparable order to the 70B slice of the Llama 3 herd.²)
(c) MFU from 400 TFLOP/s. $400/989\approx$ 40.4% MFU — right on target. Wall-clock: 4.4M GPU-hours / 512 GPUs ≈ ~8,600 hours / 512 … careful: 4.4M / 512 ≈ 8,600 GPU-hours per GPU → /24 ≈ ~360 days. That’s too long — 512 GPUs is undersized for 70B×15T; you’d want several thousand GPUs (e.g. 4,096 → ~45 days). The estimate itself is the deliverable: it tells you the cluster is too small before you waste a run.

Practice problem 2: FP8 economics and the recompute/precision budget

Problem. (a) DeepSeek-V3 reports 2.788M H800 GPU-hours for 671B total / 37B active params on 14.8T tokens. Sanity-check the pre-training compute against $C\approx 6N_{\text{active}}D$ and comment. (b) You’re memory-limited and considering full recompute (30–40% overhead) vs FP8 (2× GEMM throughput). Which helps throughput, which helps memory, and how do they interact with MFU?

Worked solution.
(a) MoE cost scales with activated params, not total: $C\approx 6\times 37\times 10^9\times 14.8\times 10^{12}\approx 3.3\times 10^{24}$ FLOPs. Pre-training was 2.664M GPU-hours = $2.664\times 10^6\times 3600\approx 9.6\times 10^9$ GPU-s. Implied effective throughput $=3.3\times 10^{24}/9.6\times 10^9\approx 3.4\times 10^{14}$ FLOP/s/GPU ≈ 340 TFLOP/s — plausible for FP8 on H800 with real-world overheads, and the low denominator (37B active, not 671B) is why the MoE was so cheap.¹ The lesson: MoE decouples capacity from compute — you pay attention+active-expert FLOPs, not total-param FLOPs.
(b) Full recompute helps memory, hurts throughput: it frees activation memory (down to $2sbh$/layer) but adds 30–40% FLOPs that count against MFU. FP8 helps throughput (≈2× on the three GEMMs) and memory (FP8 activations/weights are half the bytes). They compose: FP8 buys back some of the headroom recompute costs, and can make full recompute unnecessary if FP8’s memory savings alone let you fit. Interaction with MFU: FP8 raises absolute TFLOP/s; full recompute lowers MFU (extra non-model FLOPs). The optimal point is usually selective recompute + SP + FP8 — fit the model with the cheapest memory tool and spend the precision win on speed.⁸¹

---

13. Practice & resources

Hands-on

1. Run a small multi-node pretrain in TorchTitan. Clone pytorch/torchtitan, train a small Llama config on 2 nodes, and toggle one dimension at a time — TP, PP, FSDP2, torch.compile, Float8 — watching tokens/sec and peak memory. This is the fastest way to feel the tradeoffs in §4–§6. github.com/pytorch/torchtitan
2. Configure Megatron-Core parallelism. Set --tensor-model-parallel-size, --pipeline-model-parallel-size, --context-parallel-size, --expert-model-parallel-size, --sequence-parallel, and --recompute-granularity selective. Sweep TP∈{1,2,4,8} on one node and confirm TP>8 (crossing the NVLink boundary) tanks throughput — the §3 constraint, measured. github.com/NVIDIA/Megatron-LM
3. Read the whole 3D stack in nanotron/picotron. For understanding, huggingface/nanotron and picotron implement DP+TP+PP+EP in readable PyTorch. Trace one forward/backward and find where each collective fires. github.com/huggingface/nanotron
4. Turn on FP8 with Transformer Engine. Wrap a model in te.fp8_autocast with DelayedScaling, compare loss curves and step time vs BF16, then inspect the amax history. See where instability appears if you FP8 the wrong ops. github.com/NVIDIA/TransformerEngine
5. Async-checkpoint a run. Use torch.distributed.checkpoint.async_save; measure the GPU stall (D2H copy) vs the background write, and reload under a different parallelism layout to see DCP’s resharding. pytorch.org DCP recipe

Real resources

• The Ultra-Scale Playbook (Hugging Face) — the single best end-to-end tour of 5D parallelism, memory math, and overlap, with interactive plots. Start here. huggingface.co/spaces/nanotron/ultrascale-playbook
• Llama 3 paper §3.3 — infrastructure, the [TP,CP,PP,DP] config table, MFU, and the reliability section. arXiv:2407.21783
• DeepSeek-V3 Technical Report §3 — DualPipe, FP8 recipe, EP, custom comms. The reference for FP8-at-scale. arXiv:2412.19437
• Reducing Activation Recomputation (Korthikanti et al.) — the activation-memory formula, selective recompute, sequence parallelism. arXiv:2205.05198
• Megatron-LM paper(s) — TP (arXiv:1909.08053) and pipeline/3D-parallelism scaling (arXiv:2104.04473).
• stas00/ml-engineering — the practitioner’s field guide to distributed training, MFU, debugging, and hardware. github.com/stas00/ml-engineering

---

14. What’s next

Two threads, in order:

1. 12-llm-inference-systems (next) — training’s mirror image. Inference is memory-bandwidth-bound on the KV cache and latency-sensitive, with a whole different bag of tricks (paged attention, continuous batching, tensor/pipeline parallelism for serving, speculative decoding, disaggregated prefill/decode). Everything you learned about the memory wall and interconnect ladder reappears with a decode-shaped twist.
2. 10-model-parallelism (back) — reread TP/PP/EP/CP now that you’ve seen them composed. The dimensions make more sense once you’ve watched them fight over the same interconnect budget.

Further threads: MoE systems (expert-parallel routing, load balancing, EPLB), low-precision frontiers (MXFP8/FP4 training numerics), and RL/post-training infrastructure (asynchronous rollout+train systems, which Llama 4 and the o-series pushed hard).

---

Reference topic: gpu-systems-for-llms | Concepts: 3d-parallelism, recomputation, fp8-training, megatron-lm, mfu | Filed: 2026-09-02

---

References

Footnotes

1. [recent] DeepSeek-AI, “DeepSeek-V3 Technical Report,” arXiv:2412.19437 (2024). arxiv.org/abs/2412.19437. §1/Table 1: 2.788M H800 GPU-hours total (2.664M pre-train + 0.119M ctx-ext + 0.005M post-train), **$5.576M\ast \ast @$2/GPU-hr, 14.8T tokens. §2: 671B total / 37B active, 256 routed + 1 shared experts, top-8, ≤4 nodes/token, MLA. §3.1–3.2: 2,048 H800 (8/node, NVLink+NVSwitch intra, IB inter), 16-way PP, 64-way EP, ZeRO-1 DP, no TP; DualPipe bidirectional PP hiding all-to-all; custom PTX all-to-all kernels, 20 SMs / 10 comm channels. §3.3 FP8: activations tile-wise 1×128, weights block-wise 128×128, all-E4M3, FP8 on Fprop/Dgrad/Wgrad only; embeddings/output-head/gating/norm/attention in BF16/FP32; FP32 master weights + gradients, BF16 AdamW moments; FP32 CUDA-core accumulation every N_C=128; relative error vs BF16 <0.25%. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹ ↩¹⁰ ↩¹¹ ↩¹² ↩¹³ ↩¹⁴ ↩¹⁵

2. [recent] Meta, Llama 3.1 Model Card (GitHub, meta-llama/llama-models). github.com/meta-llama/llama-models. 405B: 30.84M H100-80GB GPU-hours (herd total 39.3M); total pre-training compute 3.8×10²⁵ FLOPs. ↩ ↩²

3. [recent] Grattafiori et al. (Meta), “The Llama 3 Herd of Models,” arXiv:2407.21783 (2024). arxiv.org/abs/2407.21783. §3.3: 16,384 H100s; 4D parallelism [TP, CP, PP, DP] with config table (TP=8, CP=1, PP=16, DP=128 at 8K ctx → TP=8, CP=16, PP=16, DP=8 at 128K); BF16 MFU 38–43% (43% @ 8,192 GPUs, 41% @ 16,384, 38% @ 131K ctx); ordering “optimized for network communication,” DP outermost as most latency-tolerant; 405B trained in BF16 (FP8 only for inference); RoCE fabric for 405B, InfiniBand for smaller models, 400 Gb/s + NVLink intra-node; 466 interruptions over 54 days (419 unexpected, ~78% hardware, GPU largest category), >90% effective training time, ~3 manual interventions total. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹ ↩¹⁰ ↩¹¹ ↩¹² ↩¹³

4. [recent] Meta, “The Llama 4 herd,” AI-at-Meta blog (April 2025). ai.meta.com/blog/llama-4-multimodal-intelligence. Behemoth pre-trained in FP8 on 32K GPUs at ~390 TFLOP/s/GPU (~19.7% of ~1,979 FP8 peak); 128-expert MoE; FP8 the first at Meta for pre-training (following DeepSeek-V3). MoE FP8-instability framing corroborated by secondary analysis (Cameron Wolfe, “Llama 4,” cameronrwolfe.substack.com). Reported figures; exact FP8 scheme not disclosed in detail. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹

5. NVIDIA H100 interconnect specs (NVLink 4 / NVSwitch 3 / InfiniBand NDR). NVLink 4: 900 GB/s bidirectional per GPU (18 links × 50 GB/s bidir; ≈450 GB/s each way) via 4 NVSwitch chips per HGX/DGX node; InfiniBand NDR 400 Gb/s ≈ 50 GB/s per GPU (1 ConnectX-7 NIC/GPU) → intra:inter ≈ 18×; SHARP in-network reduction. NVIDIA HGX H100 / DGX H100 documentation and NVIDIA Hopper Architecture In-Depth blog: developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth. ↩ ↩² ↩³

6. NVIDIA, “NVIDIA H100 Tensor Core GPU Architecture” whitepaper — 80 GB HBM3, 989 TFLOP/s BF16 (dense) / ~1,979 TFLOP/s FP8 tensor-core peak; 900 GB/s NVLink; used for the peak-FLOP denominators in the MFU calculations above. ↩

7. [recent] DeepSeek-AI, “Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures,” arXiv:2505.09343 (2025). arxiv.org/abs/2505.09343. Frames H800 NVLink ~200 GB/s peak (~160 GB/s achieved) vs 400 Gbps IB NIC ~50 GB/s (~40 GB/s effective), ~4:1 scale-up:scale-out ratio; motivates no-TP + FP8 + EP design. Note: NVLink bandwidth is quoted inconsistently across sources (spec vs bidirectional vs achieved) — see GitHub deepseek-ai/DeepSeek-V3 issue #479. ↩

8. [established] Korthikanti et al. (NVIDIA), “Reducing Activation Recomputation in Large Transformer Models,” arXiv:2205.05198 (MLSys 2023). arxiv.org/abs/2205.05198. Activation memory per layer $=sbh(34+5as/h)$ bytes; selective recompute drops the $5as/h$ attention term (→$34sbh$, ~2–7% overhead); full recompute →$2sbh$ (~30–40% overhead); sequence parallelism divides by TP degree $t$ (→$34sbhL/t$); combined ~5× activation-memory reduction, >90% recompute-overhead cut. 530B: 54.2% MFU (SP+selective) vs 42.1% (full recompute); 1T: 56.3% MFU / 57.0% HFU. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸

9. [established] Chowdhery et al. (Google), “PaLM: Scaling Language Modeling with Pathways,” arXiv:2204.02311 (2022). arxiv.org/abs/2204.02311. Defines MFU (model FLOPs / peak, excludes recomputation, ~6N accounting) vs HFU (all executed FLOPs, includes recomputation; HFU≥MFU). PaLM 540B: 46.2% MFU, 57.8% HFU. ↩ ↩² ↩³ ↩⁴

10. [recent] NVIDIA, Transformer Engine (docs + GitHub). github.com/NVIDIA/TransformerEngine. FP8/FP4 training/inference on Hopper/Ada/Blackwell; fp8_autocast; recipes DelayedScaling, per-tensor current scaling, MXFP8 block scaling (Blackwell-only); backend for Megatron-Core and NeMo. [established] for the core library; MXFP8 [recent]. ↩ ↩² ↩³ ↩⁴ ↩⁵

11. [established] PyTorch FSDP2 (fully_shard) — per-parameter DTensor sharding (FSDP1 deprecated); overlaps reduce-scatter/all-gather with compute; gradient bucketing in DDP amortizes latency. Zhao et al., “PyTorch FSDP,” arXiv:2304.11277 (2023); FSDP2 tutorial: docs.pytorch.org/tutorials/intermediate/FSDP_tutorial.html. ↩ ↩² ↩³

12. [established] NVIDIA Megatron-LM / Megatron-Core. github.com/NVIDIA/Megatron-LM, developer.nvidia.com/megatron-core. Composable TP/PP/EP/CP + sequence parallelism library; FP8/FP4 via TE. Foundational papers: Shoeybi et al. arXiv:1909.08053 (TP); Narayanan et al. arXiv:2104.04473 (3D-parallel scaling). Development moved to open GitHub CI in Dec 2025 (repo README). [recent] for the exact version/status. ↩ ↩²

13. [established] Microsoft DeepSpeed. github.com/microsoft/DeepSpeed. ZeRO stages 1/2/3 (Rajbhandari et al., arXiv:1910.02054), ZeRO-Offload/Infinity (arXiv:2104.07857), ZeRO++. ↩ ↩²

14. [recent] Liang et al. (PyTorch), “TorchTitan: One-stop PyTorch native solution for production ready LLM pre-training,” arXiv:2410.06511 (ICLR 2025). arxiv.org/abs/2410.06511, github.com/pytorch/torchtitan. FSDP2, TP (incl. async-TP), PP, CP → 3D/4D; torch.compile; Float8; evaluated on Llama 3.1 8B–405B. ↩ ↩²

15. [recent] NVIDIA NeMo Framework. github.com/NVIDIA-NeMo, github.com/NVIDIA/NeMo. End-to-end (pretrain/post-train/RL/multimodal/speech) built on Megatron-Core via Megatron-Bridge + Transformer Engine; reorganized into a multi-repo NVIDIA-NeMo org and shifting to PyTorch-native loops (Megatron-Bridge, NeMo AutoModel, NeMo-RL). [recent] — mid-restructuring as of 2026. ↩ ↩²

16. [recent] Hugging Face nanotron. github.com/huggingface/nanotron. Minimal 3D-parallelism (DP+TP+PP, +EP) library; trained SmolLM3 (384 H100s, 24 days — huggingface.co/blog/smollm3); paired with educational picotron and the Ultra-Scale Playbook. ↩ ↩²

17. [recent] Stas Bekman, “Machine Learning Engineering” (ml-engineering). github.com/stas00/ml-engineering. Practitioner reference for the “good MFU 35–50%, 50%+ excellent” rule of thumb and distributed-training debugging. [contested]/community-consensus — not a single primary-source number. ↩

18. [recent] Kokolis et al. (Meta), “Revisiting Reliability in Large-Scale Machine Learning Research Clusters,” arXiv:2410.21680 (2024). arxiv.org/abs/2410.21680. Failure-model projection: MTTF ~47 days @ 8 GPUs → ~8 h @ 1,024 → ~1.8 h @ 16,384 GPUs; “lemon node” detection cut 512+-GPU job failure rate 14%→4%; introduces ETTR. Paper uses MTTF; secondary sources often say MTBF. ↩

19. [recent] PyTorch Distributed Checkpoint (DCP) + async save. docs.pytorch.org/tutorials/recipes/distributed_async_checkpoint_recipe.html, pytorch.org/blog/distributed-checkpoint-efficient-checkpointing-in-large-scale-jobs. Sharded, reshardable parallel checkpoint; async_save stages to CPU then writes on a background thread (GPU blocked only for the D2H copy). ↩

20. [established] Wang et al., “GEMINI: Fast Failure Recovery in Distributed Training with In-Memory Checkpoints,” SOSP 2023. cs.rice.edu/~eugeneng/papers/SOSP23.pdf. Checkpoints to CPU DRAM + peer-machine memory; >100× faster checkpoint/recovery, per-iteration checkpointing; near-optimal placement + traffic scheduling. ↩

21. [established] Mohan, Phanishayee, Chidambaram, “CheckFreq: Frequent, Fine-Grained DNN Checkpointing,” USENIX FAST 2021. usenix.org/conference/fast21/presentation/mohan. Two-phase snapshot()+persist() pipelined with compute; adaptive frequency bounds overhead to ~3.5%. ↩