Expert Parallelism (EP)

Definition

Expert parallelism places the experts of a Mixture-of-Experts (MoE) layer on different GPUs. MoE replaces a dense FFN with $E$ expert FFNs + a router that sends each token to its top-$k$ experts ($k=1$ or $2$), so parameters grow with $E$ while per-token FLOPs stay ~constant (conditional compute). Since experts hold most of the parameters and are too big to replicate, EP splits them across GPUs and shuffles tokens with two all-to-all collectives per layer.

Key math / How it works

A token arrives on the GPU holding its sequence position but must be computed by the GPU holding its chosen expert — an arbitrary shuffle. Per MoE layer:

1. Router picks top-$k$ experts per token.
2. All-to-all DISPATCH — route each token to the GPU holding its expert.
3. Expert GEMMs compute locally.
4. All-to-all COMBINE — send outputs back to each token’s home position.

Why all-to-all is the hard part. Unlike all-reduce (per-GPU volume independent of parallelism degree), all-to-all volume $\propto$ tokens × hidden, and as EP grows the shuffle spills cross-node onto scarce IB bandwidth. Fine-grained MoE compounds this: many small experts + many MoE layers (DeepSeek-V3: ~58 MoE layers, 256 routed + 1 shared expert, two all-to-alls each) mean high frequency, cross-node latency, and low arithmetic intensity (small experts finish fast, leaving little compute to hide comms behind). Megatron reports comms can exceed 50% of training time at DeepSeek-V3 scale without optimization.

Mitigations (2025–26): DeepEP (NVSHMEM-based dispatch/combine kernels) and NVIDIA HybridEP overlap all-to-all with expert GEMMs near hardware-limit bandwidth; overlapping the shared-expert MLP with dispatch; “1F1B A2A overlap” hiding EP comms inside the pipeline schedule (DeepSeek’s DualPipe).

Load balancing. The router can collapse onto a few popular experts, starving the rest and creating stragglers (the all-to-all waits for the slowest expert). Levers:

• Auxiliary load-balancing loss (Shazeer et al., 2017) — penalize uneven assignment; DeepSeek-V3 moves to an auxiliary-loss-free bias-adjustment scheme to avoid the aux-loss quality tax.
• Capacity factor + token dropping — cap tokens per expert; overflow tokens skip the expert.

Orthogonal axis. Experts are replicated across the DP dim and split across the EP dim; you can combine EP with TP inside each expert (Megatron exposes expert-model-parallel-size and expert-tensor-parallel-size separately).

Why it matters

EP is what makes trillion-parameter MoEs trainable: it decouples model capacity from per-token compute, so a 671B model costs only its ~37B active params in FLOPs. It is the primary intra-expert sharding axis for MoE (DeepSeek-V3: 64-way EP, no TP), and its all-to-all is the defining communication challenge of modern frontier MoE training.

Taught in

• 10-model-parallelism — §7 the MoE all-to-all, load balancing.
• 11-sota-large-scale-training — DeepSeek-V3 EP recipe.

See also

• tensor-parallelism — composes inside each expert
• pipeline-parallelism — DualPipe hides the all-to-all
• collective-communication — all-to-all dispatch/combine
• 3d-parallelism — EP as a distinct axis
• fp8-training — MoE is the most FP8-sensitive architecture
• gpu-systems-for-llms