Data Parallelism (DP)

Definition

Data parallelism is the baseline axis of distributed training: replicate the full model (weights + gradients + optimizer state) on every GPU, split the global batch across the $N$ replicas, run identical forward/backward on each local shard, then average the gradients with one all-reduce before a lockstep optimizer step. It is embarrassingly parallel on the compute side with exactly one synchronization point per step. PyTorch’s DistributedDataParallel (DDP) is the canonical implementation.

Key math / How it works

On $N$ ranks: rank $i$ computes $g_i=\frac{1}{B/N}{\sum }_{x\in {\text{batch}}_i}{\nabla }_{\theta }\ell (x)$; all ranks then obtain
$\bar{g}=\frac{1}{N}{\sum }_{i=0}^{N-1}{g}_i,$
which is mathematically identical to the full global-batch gradient — that equivalence is the whole point. Since weights, optimizer state, and $\bar{g}$ stay identical across ranks, replicas remain bit-for-bit synchronized; only gradients are ever communicated, never weights.

Global (effective) batch size:
$B_{\text{global}}=B_{\text{local}}\times N\times A_{\text{accum}}$
This — not per-GPU batch — governs optimization dynamics. Scaling GPUs silently changes it; the linear LR scaling rule (multiply LR by $k$ when batch $\times k$, with warmup) compensates (Goyal et al., ResNet-50 @ batch 8192).

Gradient accumulation: split the local batch into $A_{\text{accum}}$ micro-batches, sum .grad across them, step only after the last. Activation memory is set by the micro-batch, so accumulation emulates a large batch at small memory cost. Crucially, skip the all-reduce on non-final micro-batches (no_sync() in DDP) — one network sync per optimizer step, not per micro-batch.

DDP performance: gradients coalesce into buckets (default 25 MB), filled in reverse model.parameters() order; each bucket’s all-reduce launches asynchronously via an autograd hook the moment its grads are ready, overlapping with the still-computing earlier layers. Ideally only the first-layer bucket is left “exposed.” Step time $\approx \max (T_{\text{compute}},T_{\text{comm}})$ with perfect overlap.

The wall: DP shards the data, never the model. Every replica holds the full 16 bytes/param of state — a 7B model is 112 GB, already over an 80 GB H100. No amount of DP fixes this; adding GPUs adds identical copies. This is the entire motivation for zero/fsdp.

Why it matters

DP is the first, simplest, and still most-used parallelism axis; it is the outermost (“most latency-tolerant”) leg of every 3D-parallel frontier run because its single gradient sync overlaps cleanly with backward. Understanding when it becomes comms-bound (small per-GPU batch, slow fabric) or memory-bound (model state exceeds one GPU) is the pivot point to every sharding technique.

Taught in

• 08-data-parallelism-and-collectives — §3 mechanics, §8 DDP, §9 the memory wall.

See also

• collective-communication — the all-reduce that averages gradients
• zero / fsdp — DP that also shards the model state
• optimizer-state-memory — the 16 B/param DP must replicate
• nccl — the collective backend DDP calls
• 3d-parallelism — DP as the outermost axis
• gpu-systems-for-llms