Pipeline Parallelism (PP)

Definition

Pipeline parallelism splits the model across layers: assign layers $\mathrm{1..}L/p$ to stage 0, the next chunk to stage 1, etc., on $p$ GPU groups. Communication is only the activation at stage boundaries, sent GPU→GPU with cheap point-to-point send/recv (no collective) — so PP tolerates the slower inter-node InfiniBand link, unlike TP. The cost is a new enemy: the pipeline bubble.

Key math / How it works

Naively, stage $i$ can’t start until stage $i-1$ finishes the whole batch → only one GPU works at a time ($p\times$ slowdown). Micro-batching (GPipe, Huang et al., arXiv:1811.06965) splits the batch into $m$ micro-batches streamed through so stages overlap. But the pipe still fills (stages $\mathrm{2..}p$ idle while microbatch 1 propagates) and drains — that idle time is the bubble.

Deriving the bubble fraction. With per-microbatch times $t_f,t_b$ on one stage: useful work $=m(t_f+t_b)$; bubble (fill $p-1$ + drain $p-1$) $=(p-1)(t_f+t_b)$:
$\boxed{\text{bubble fraction}=\frac{p-1}{m+p-1}}$
For $m\gg p$, $\approx \frac{p-1}{m}$. The ratio is what matters: $p=8,m=16$ and $p=16,m=32$ both give $\approx 30\%$. Shrink it by raising $m$ — capped by memory and global batch. (Example: $p=8$, want $\le 15\%$ → need $m\ge 40$.)

Schedules:

• GPipe — all forwards, then all backwards. Simple, but stores activations for all $m$ microbatches → $O(m)$ activation memory.
• 1F1B (PipeDream, Narayanan et al., arXiv:1806.03377**)** — after a $p-1$ warmup, each stage alternates one-forward-one-backward, freeing a microbatch’s activations right after its backward → $O(p)$ memory. Same bubble as GPipe (common misconception: 1F1B does not reduce the bubble — it’s a memory optimization).
• Interleaved 1F1B (Megatron, arXiv:2104.04473**)** — each device gets $v$ non-contiguous layer chunks (“virtual stages”), shrinking fill/drain per chunk:
  ${\text{bubble}}_{\text{interleaved}}=\frac{1}{v}\cdot \frac{p-1}{m}$
  Price: $v\times$ more P2P traffic (each microbatch crosses stage boundaries $v\times$ as often); Megatron routes the extra reassembly over NVLink. Constraint: $m$ a multiple of $p$.
• Zero-Bubble PP (Qi et al., 2024) [recent] — splits backward into input-grad (B) and weight-grad (W); W has no downstream dependency, so it fills the bubble, driving it toward zero.

Why it matters

PP cuts per-GPU parameter memory with cheap, latency-tolerant P2P comms, making it the axis that spans nodes in N-D parallelism (Llama 3: PP=16; DeepSeek-V3: 16-way PP with the DualPipe bidirectional schedule hiding MoE all-to-all). The bubble is the tax you pay, amortized by many microbatches or interleaving.

Taught in

• 10-model-parallelism — §5 bubble derivation, GPipe/1F1B/interleaved.

See also

• tensor-parallelism — the within-layer alternative (NVLink-bound)
• megatron-lm — interleaved 1F1B + 3D scaling
• collective-communication — PP uses P2P, not collectives
• 3d-parallelism — PP as the across-nodes axis
• recomputation — the other activation-memory lever
• gpu-systems-for-llms