Megatron-LM / Megatron-Core

Definition

Megatron-LM (NVIDIA) is the reference framework and body of techniques for large-scale transformer training on GPU clusters. Its two foundational papers introduced tensor parallelism (Shoeybi et al., arXiv:1909.08053) and the 3D-parallel scaling analysis (Narayanan et al., arXiv:2104.04473). Megatron-Core is the composable library (layers, parallelism, optimizers) that others — NeMo, and much of the field — build on; Megatron-LM is the example training stack.

Key math / How it works

Megatron is where most of N-D parallelism originated or was made practical:

• Tensor parallelism — the column→row GEMM pairing so a two-GEMM block needs only one all-reduce (formalized as conjugate operators $f$/$g$: $f$ = identity fwd / all-reduce bwd, $g$ = all-reduce fwd / identity bwd), and head-split multi-head attention. Constraint: hidden_size and num_attention_heads divisible by the TP degree.
• Sequence parallelism (Korthikanti et al., arXiv:2205.05198) — split LN/dropout/residual along the sequence, interleaved with TP at no extra bandwidth; enable whenever TP > 1.
• Pipeline parallelism — GPipe / 1F1B / interleaved 1F1B (virtual pipeline stages cut the bubble by $\sim v$ at $v\times$ P2P cost); the bubble-fraction analysis $\frac{p-1}{m+p-1}$.
• Expert parallelism — MoE all-to-all dispatch/combine, with expert-model-parallel-size and expert-tensor-parallel-size exposed separately; DeepEP/HybridEP overlap.
• Context parallelism, activation recomputation (selective + full), and FP8 via Transformer Engine.

The topology guidance Megatron codified: TP within a node (≤8, NVLink), PP across nodes (IB-tolerant P2P), DP outermost (overlapped grad sync); rank ordering TP-innermost → CP/EP → PP → DP-outermost. Practical config flags: --tensor-model-parallel-size, --pipeline-model-parallel-size, --context-parallel-size, --expert-model-parallel-size, --sequence-parallel, --num-layers-per-virtual-pipeline-stage, --recompute-granularity selective.

Landscape (Sept 2026): Megatron-Core is the most battle-tested TP/PP/EP/CP stack on NVIDIA hardware; alternatives are DeepSpeed (home of ZeRO), TorchTitan (PyTorch-native FSDP2+TP+PP+CP+Float8), NeMo (end-to-end, built on Megatron-Core), and nanotron/picotron (minimal, readable, for learning).

Why it matters

Nearly every published frontier training recipe traces its parallelism techniques to Megatron: the column→row TP pairing, the pipeline bubble formula, sequence parallelism, and interleaved 1F1B are all Megatron contributions now considered standard. Reach for Megatron-Core (or NeMo on top) to train a huge dense or MoE model on NVIDIA hardware; it is the practical embodiment of the entire gpu-systems-for-llms toolbox.

Taught in

• 10-model-parallelism — TP/SP/PP/EP mechanics and Megatron code.
• 11-sota-large-scale-training — §8 framework landscape.

See also

• tensor-parallelism, pipeline-parallelism, sequence-parallelism, expert-parallelism, context-parallelism — the techniques it introduced/refined
• 3d-parallelism — the composition Megatron scales
• fp8-training — via Transformer Engine
• zero / fsdp — the DP-sharding alternatives it composes with
• gpu-systems-for-llms