Sequence Parallelism (SP)

Definition

Sequence parallelism (Korthikanti et al., arXiv:2205.05198) splits the activation-heavy but non-tensor-parallel regions of a transformer block — LayerNorm, dropout, and residual add — along the sequence dimension across the TP group. In vanilla Megatron TP these operate on the full $bsh$ activation replicated identically on all $t$ ranks — pure redundancy that dominates activation memory at scale. SP interleaves with TP to remove it at zero extra bandwidth.

Key math / How it works

LayerNorm and dropout are per-token operations, so they parallelize trivially over the sequence: rank $i$ owns tokens $s\cdot i/t:s\cdot (i+1)/t$, with no communication within the region. Activation memory for these regions drops by $t\times$.

The splice into TP at block boundaries (the conjugate operators):

• Entering the TP block ($f$): sequence-sharded → hidden-replicated = all-gather (fwd) / reduce-scatter (bwd).
• Leaving the TP block ($g$, previously a single all-reduce): hidden-partial → sequence-sharded = reduce-scatter (fwd) / all-gather (bwd).

Why it’s free. A ring all-reduce is a reduce-scatter followed by an all-gather. SP replaces TP’s single all-reduce ($g$) with a reduce-scatter and TP’s identity ($f$) with an all-gather — the same bytes on the wire, just re-associated so that between the reduce-scatter and the next all-gather each GPU holds only $1/t$ of the activation, with the LayerNorm/dropout/residual sitting in between. This is why modern Megatron enables SP whenever TP > 1.

Combined memory effect (with selective recompute), across $L$ layers:
${\text{mem}}_{\text{act}}^{\text{SP+selective}}=\frac{34sbhL}{t}$
i.e. the whole activation term divides by the TP degree. Megatron reported SP + selective recompute delivering ~5× activation-memory reduction and >90% cut in recompute overhead vs. full recompute (530B: 54.2% MFU with SP+selective vs 42.1% with full recompute).

Why it matters

SP is a nearly-free multiplier on activation memory that lets you push sequence length and batch size without OOM, and it is enabled by default alongside TP in every modern N-D parallel recipe. It attacks the linear $34sbh$ activation term; for the quadratic attention term at long context you add context parallelism.

Taught in

• 10-model-parallelism — §4 sequence parallelism.
• 11-sota-large-scale-training — §5 activation memory & recompute.

See also

• tensor-parallelism — SP is fused with TP
• context-parallelism — splits the sequence for the attention term
• recomputation — the complementary activation lever
• megatron-lm — where SP is implemented
• collective-communication — RS + AG re-association
• gpu-systems-for-llms