3D Parallelism (DP × TP × PP)

Definition

3D parallelism composes the three orthogonal axes — data × tensor × pipeline parallelism — into one training configuration, extended to 4D/5D with expert (EP) and context (CP) parallelism for MoE and long context. No single axis scales alone; the art is mapping each onto the interconnect fast enough for its traffic while the model fits in aggregate memory.

Key math / How it works

$\text{total GPUs}=\underset{\text{DP}}{\underbrace{d}}\times \underset{\text{TP}}{\underbrace{t}}\times \underset{\text{PP}}{\underbrace{p}}[\times \underset{\text{EP}}{\underbrace{e}}\times \underset{\text{CP}}{\underbrace{c}}]$

Each axis has a distinct comms profile that dictates where on the topology it must live (the ~18× NVLink-vs-IB cliff on H100 — 900 vs ~50 GB/s):

Axis	Splits	Comms	Placement
TP	each matmul	all-reduce every layer (critical path)	inside a node, NVLink, $t\le 8$
SP	LN/dropout/residual (seq)	AG/RS, fused with TP	inside node
CP	sequence (attention)	ring P2P, overlapped	fast links
PP	layers → stages	P2P at boundaries	across nodes (IB-tolerant)
EP	MoE experts	all-to-all	node / cross-node
DP	the batch	grad sync once/step, overlapped	outermost (most latency-tolerant)

The placement rule: put the chattiest dimension on the fastest link. Canonical rank ordering (Llama 3): [TP, CP, PP, DP] innermost→outermost. TP innermost so its per-layer all-reduces ride NVLink; DP outermost because its single overlapped gradient sync tolerates the most latency.

How to reason about a config — memory-first, then comms:

1. Compute per-GPU memory need (params/shard + optimizer + peak activations).
2. Params don’t fit → add PP (cheap comms) until they do.
3. A layer’s activations don’t fit / want higher per-GPU FLOPs → add TP (≤8, NVLink) + SP.
4. MoE experts too big → EP. Sequence too long → CP.
5. Fill remaining GPUs with DP (ZeRO/FSDP) for throughput; size microbatches to shrink the bubble.

Real configs:

Run	GPUs	TP	CP	PP	EP	DP
Llama 3 405B (8K)	16,384	8	1	16	—	128
Llama 3 405B (128K)	16,384	8	16	16	—	8
DeepSeek-V3	2,048	0	1	16	64	ZeRO-1

Why it matters

3D/4D/5D parallelism is how frontier models are actually trained. A naive config can leave half the cluster idle (Llama 4 Behemoth ran ~19.7% of FP8 peak on 32K GPUs); a good one keeps each axis’s collective on a link that can afford it and overlaps everything. This is the core reasoning of a training-systems engineer — the toolbox from which every large run is assembled.

Taught in

• 10-model-parallelism — §8 composing everything, topology mapping.
• 11-sota-large-scale-training — §4 choosing a config, real case studies.

See also

• data-parallelism, tensor-parallelism, pipeline-parallelism — the three core axes
• expert-parallelism, context-parallelism, sequence-parallelism — the 4D/5D extensions
• zero / fsdp — the DP leg
• megatron-lm — the reference framework
• mfu — the efficiency metric a config targets
• gpu-systems-for-llms