GRPO (Group Relative Policy Optimization)

Definition

Group Relative Policy Optimization (DeepSeekMath, Shao et al. 2024; arXiv:2402.03300) is PPO with the value network deleted. It replaces GAE’s learned per-state baseline with a Monte-Carlo group baseline: for each prompt, sample a group of $G$ complete outputs, score them, and use the group’s mean reward as the baseline. It is the RL algorithm behind DeepSeek-R1, exploiting the fact that reasoning rewards are sparse and terminal (one scalar per whole output) — so there is no per-token reward to bootstrap a critic from anyway.

Key math

For a group $\{o_i{\}}_{i=1}^G\sim {\pi }_{{\theta }_{\text{old}}}(\cdot \mid q)$ with rewards $\mathbf{r}=(r_1,\dots ,r_G)$, the group-relative advantage standardizes within the group and broadcasts the same scalar to every token of output $i$:
${\hat{A}}_{i,t}=\frac{r_i-\mathrm{mean}(\mathbf{r})}{\mathrm{std}(\mathbf{r})}.$
The group mean is a valid baseline (depends on $q$, not on which output’s gradient is taken → unbiased). Keep PPO’s clipped surrogate and add an explicit KL term (via Schulman’s unbiased, $\ge 0$ $k_3$ estimator):
${\mathcal{J}}_{\text{GRPO}}(\theta )=\mathbb{E}\left[\frac{1}{G}{\sum }_i\frac{1}{|o_i|}{\sum }_t\right(\min ({\rho }_{i,t}{\hat{A}}_{i,t},\mathrm{clip}({\rho }_{i,t},1-ϵ,1+ϵ){\hat{A}}_{i,t})-\beta {\mathbb{D}}_{\text{KL}}[{\pi }_{\theta }\Vert {\pi }_{\text{ref}}]\left)\right],$
with ${\rho }_{i,t}={\pi }_{\theta }(o_{i,t}\mid q,o_{i,<t})/{\pi }_{{\theta }_{\text{old}}}(\cdot )$. KL placement differs from RLHF-PPO: it is a separate loss term, not folded into the per-token reward — and RLVR recipes often set $\beta$ small or zero. Variance is controlled by group size $G$ ($\sim 1/G$; DeepSeekMath used $G=64$), traded against extra rollout compute.

Normalization biases (Dr. GRPO)

Both normalizers inject optimization bias (Liu et al. 2025, Dr. GRPO; arXiv:2503.20783):

• Response-length bias (from $1/|o_i|$): dilutes the per-token penalty on long wrong answers → pushes wrong answers longer (length inflation).
• Question-difficulty bias (from $1/\mathrm{std}(\mathbf{r})$): up-weights low-variance (very easy/hard) prompts, distorting the curriculum.

Dr. GRPO removes both (constant token normalizer; drop the std division, use $r_i-\mathrm{mean}$). Degenerate case: if all $G$ rewards are equal (all right/wrong), $\mathrm{std}=0$ and the advantage vanishes → no gradient; DAPO’s dynamic sampling (arXiv:2503.14476) filters these out. DAPO also adds clip-higher (anti-entropy-collapse), token-level loss, and overlong reward shaping; GSPO (arXiv:2507.18071) moves the ratio/clipping to the sequence level (stabilizes MoE RL, used for Qwen3).

Why it matters

GRPO deletes one of PPO’s two trainable models and all of GAE’s machinery, making critic-free RLVR tractable at scale — the workhorse for reasoning RL. Combined with verifiable rewards on a base model (no SFT), it produced emergent long chain-of-thought (R1-Zero). The value-based comeback (VAPO/VC-PPO, arXiv:2504.05118) argues a well-tuned critic still wins on long CoT — so “is the critic worth it?” is mid-swing.

Taught in

• 09-rl-for-reasoning — deriving the group baseline, the full objective, Dr. GRPO/DAPO/VAPO/GSPO.
• 10-frontier-and-infra — GRPO’s $G$-sample rollout cost as the systems bottleneck.

See also

• ppo — the algorithm GRPO strips the critic from
• rlvr — the verifiable-reward setting GRPO usually runs in
• advantage-function — what the group baseline estimates
• chain-of-thought-rl — the emergent behavior GRPO+RLVR produces
• rl-post-training-infra — the generation cost of $G$ samples/prompt
• rl-for-llms