09 · RL for Reasoning (RLVR, GRPO, R1)

What you're learning

How the field pivoted from learned reward models (Tier 2) to verifiable rewards, and how a critic-free RL algorithm — GRPO — plus rule-based rewards produced emergent long chain-of-thought reasoning (DeepSeek-R1-Zero). By the end you should be able to derive the group-relative advantage from first principles, explain mechanistically why RLVR largely sidesteps reward overoptimization, reconstruct the GRPO objective term by term, and reason about the 2025–2026 fixes (Dr. GRPO, DAPO, VAPO) and the open debates (does RLVR add capability or just sharpen it?).

This is the SOTA-sensitive capstone of the tier. It assumes the full PPO machinery (04-ppo), the token-level MDP and per-token KL plumbing (05-rl-on-token-sequences), and the RLHF pipeline with its central failure mode — reward overoptimization / Goodhart (06-rlhf-pipeline). RLVR and GRPO are best understood as two independent deletions from that pipeline: delete the learned reward model (→ RLVR), and delete the value network (→ GRPO). R1 combines both.

---

1. Learning map

graph TD
    A["RLHF pipeline (06):<br/>learned RM r_φ + PPO + KL-to-ref"] --> B["Failure: reward overoptimization<br/>(Goodhart on a learnable proxy)"]
    B --> C["Deletion 1: replace r_φ<br/>with a VERIFIER<br/>→ RLVR"]
    A --> D["Cost: 4 models,<br/>value net unstable/memory-heavy"]
    D --> E["Deletion 2: drop the critic<br/>→ group-relative baseline<br/>→ GRPO"]
    C --> F["Rule-based reward:<br/>answer-match / unit tests / proof-check"]
    E --> G["Â_i = (r_i − mean r)/std r<br/>Monte-Carlo group baseline"]
    F --> H["DeepSeek-R1-Zero:<br/>pure RL on base model"]
    G --> H
    H --> I["Emergent long CoT,<br/>self-verification, 'aha moment'"]
    I --> J["Readability issues<br/>→ R1 cold-start SFT + multi-stage"]
    F --> K["o1-style: RL → long internal CoT<br/>scales with test-time compute"]
    A --> L["Process vs Outcome RM<br/>PRM (2305.20050) vs ORM"]
    L --> M["R1 rejects PRM:<br/>reward hacking + step-labeling cost"]
    G --> N["2025-26 fixes:<br/>Dr. GRPO, DAPO, VAPO, GSPO"]
    I --> O["Debate: does RLVR add<br/>capability or sharpen pass@1?"]

    style C fill:#44a,color:#fff
    style E fill:#a64,color:#fff
    style H fill:#4a4,color:#fff
    style O fill:#a44,color:#fff

Prerequisites (assumed): PPO-Clip surrogate + the four-model RLHF-PPO setup (04-ppo, ppo); autoregressive-LM-as-MDP and the $\beta \mathrm{KL}({\pi }_{\theta }\Vert {\pi }_{\text{ref}})$ term (05-rl-on-token-sequences); Bradley-Terry reward modeling and the overoptimization scaling laws (06-rlhf-pipeline, reward-model); the advantage-function.

---

2. Why this matters: escaping Goodhart where ground truth exists

Recall the through-line of lesson 06: every RLHF failure is the same shape — a gap between the learned proxy $r_{\varphi }$ and true preference, widened by an optimizer that treats $r_{\varphi }$ as an adversary to exploit. Gao et al.’s scaling laws made this quantitative: proxy reward rises monotonically while gold reward peaks and falls as you spend KL budget.¹ You mitigate it (bigger RM, ensembles, KL leash) but you never escape it, because the reward is a learnable function and any learnable function has blind spots an optimizer can find.

Now ask: what if the reward were not learnable? In math, code, and formal proofs there is a checkable ground truth. “Is $x=\frac{-b\pm \sqrt{b^2-4ac}}{2a}$ the answer?” — compare to the key. “Does this function pass the unit tests?” — run them. “Is this Lean proof valid?” — run the proof checker. The reward becomes a program, not a network. This is the core move of the lesson:

The core move — RLVR

RL with Verifiable Rewards (RLVR): in domains with programmatically checkable answers, replace the learned reward model $r_{\varphi }$ with a deterministic verifier $V(x,y)\in \{0,1\}$ (or a small discrete set). There is no proxy to overfit, because there is no proxy — the reward is the ground-truth objective (up to the coverage of the verifier). This largely dissolves the §8 failure mode of lesson 06.

Two things happened in 2024–2025 at once, and it is worth separating them because they are logically independent:

1. RLVR removes the learned reward model (the Stage-2 artifact of RLHF), replacing it with a verifier. This attacks overoptimization.
2. GRPO removes the value network / critic (the second trainable model in PPO), replacing GAE with a Monte-Carlo group baseline. This attacks operational cost and value-learning instability.

DeepSeek-R1-Zero (arXiv:2501.12948) is what you get when you do both and then run pure RL on a base model with no SFT cold-start — and the surprise was that long, self-correcting chain-of-thought emerged from nothing but “get the answer right.”²

---

3. RLVR: the reward is a program, not a proxy

3.1 What “verifiable” buys you

A verifier is a function you write, not one you fit. Canonical instances:

Domain	Verifier	Reward
Math (final-answer)	Parse boxed answer, symbolic/numeric equality to key	$1$ if match else $0$
Code	Compile + run hidden unit tests	fraction of tests passed, or all-or-nothing
Formal proofs	Lean/Coq/Isabelle proof checker	$1$ if the checker accepts
Instruction-format	Regex / parser on required tags	$1$ if well-formed

The reward model in RLHF was a proxy for “a human would prefer this.” The verifier is the actual objective for these domains — correctness is not a matter of taste. This is the mechanistic reason RLVR resists overoptimization:

Why RLVR resists overoptimization (the key insight)

In RLHF, $\arg {\max }_{\pi }\mathbb{E}[r_{\varphi }]$ and $\arg {\max }_{\pi }\mathbb{E}[\text{true quality}]$ diverge once you push past the RM’s competence region — that divergence is overoptimization. With a verifier, the optimization target and the true target are (nearly) the same function, so pushing harder on the reward pushes harder on the thing you actually want. There is no learned surface to slip on. Optimizing the reward to saturation is success, not Goodhart — provided the verifier is sound.

Note a subtle consequence: RLVR often needs a much smaller (or zero) reference-KL leash than RLHF. In lesson 06 the $\beta \mathrm{KL}({\pi }_{\theta }\Vert {\pi }_{\text{ref}})$ term existed largely to keep the policy inside the region where $r_{\varphi }$ is trustworthy. A verifier is trustworthy everywhere it can adjudicate, so several R1-style recipes shrink $\beta$ aggressively or drop the KL term entirely (Dr. GRPO and DAPO both remove it — §8). The leash’s original job was to protect a proxy that no longer exists.

3.2 The limits — this is not a free lunch

RLVR's boundaries and verifier gaming

RLVR is only as good as (a) where verification exists and (b) how sound the verifier is.

• Coverage. Open-ended writing, dialogue, “helpfulness,” honesty, aesthetics — no programmatic verifier. RLVR does not replace RLHF/DPO there; it complements it (R1’s final stage still uses an LLM-judge + rule-based hybrid for non-verifiable tasks).
• Spec-gaming the verifier. The verifier is a specification, and a specification can be gamed even when it isn’t learnable:
  • Answer-extraction exploits: if the reward regex accepts any number matching the key anywhere in the output, the model can emit the answer without valid reasoning, or spam candidates.
  • Test-suite gaming (code): trivial or hard-coded solutions that pass weak/visible unit tests without generalizing (if input == known_case: return known_answer); reward-hacking the harness (reading expected output from files, exiting 0, monkey-patching the test framework).
  • False positives from the checker: a numeric-equality check that treats $\frac{1}{2}$ ≠ $0.5$, or accepts a right answer reached by wrong reasoning (the reward is on outcome, so lucky/guessed answers are rewarded).
  • Proof-assistant exploits: sorry/admit-style escape hatches, or exploiting checker bugs.
• Outcome-only credit assignment. A single $\{0,1\}$ at the end gives no signal about which step was wrong — the motivation for process rewards (§7).
  The takeaway: RLVR removes the learnable-proxy class of reward hacking, but not the specification-gaming class. Harden verifiers (hidden tests, adversarial cases, strict parsers) the way you’d harden any spec.

---

4. GRPO: deriving the critic-free objective

GRPO (Group Relative Policy Optimization), introduced in DeepSeekMath (arXiv:2402.03300), is the RL algorithm behind R1.³ It is PPO with the value network deleted. To see why that deletion is possible, start from what the critic was for.

4.1 What the critic did, and why we can drop it

In PPO the advantage is ${\hat{A}}_t=\left(\text{return-to-go}\right)-V_{\varphi }(s_t)$, estimated via GAE (03-actor-critic-and-gae, generalized-advantage-estimation). The value function $V_{\varphi }$ is a learned, per-state baseline whose only job is variance reduction — subtracting a baseline that doesn’t depend on the action leaves the policy-gradient unbiased (lesson 02) while shrinking its variance. But $V_{\varphi }$ costs a second full-size trainable model, and learning it is itself unstable (lesson 04 §8: an over-eager critic corrupts the advantages the actor depends on).

Here is the LLM-specific observation that makes the critic droppable. For a reasoning task, the reward is sparse and terminal: one scalar $r_i=V(x,o_i)$ for the whole output $o_i$, delivered at the end. There is no meaningful per-token reward to bootstrap a per-token value from. So instead of learning a state-conditioned baseline, estimate the baseline by Monte-Carlo: for a given prompt $q$, sample a group of $G$ complete outputs, score them all, and use the group’s mean reward as the baseline.

$b(q)\approx \frac{1}{G}{\sum }_{j=1}^G{r}_j,{\hat{A}}_i\approx r_i-b(q).$

This is a legitimate baseline: $b(q)$ depends on the prompt but not on which specific output we’re scoring (in the leave-it-in approximation), so subtracting it does not bias the gradient. GRPO then standardizes within the group — divides by the group std — to get a scale-free, dimensionless advantage:

$\boxed{{\hat{A}}_{i,t}={\tilde{r}}_i=\frac{r_i-\mathrm{mean}(\mathbf{r})}{\mathrm{std}(\mathbf{r})},\mathbf{r}=(r_1,\dots ,r_G)}$

and — crucially for outcome supervision — assigns this same scalar to every token $t$ of output $i$. Every token in a correct trajectory gets a positive advantage; every token in an incorrect one gets a negative advantage. That is the entire credit-assignment scheme: no GAE, no bootstrapping, no $\lambda$, no $\gamma$.

The one-line contrast with PPO

PPO: per-token advantage from a learned critic + GAE (bias–variance dial via $\lambda$). GRPO: per-sequence advantage from a Monte-Carlo group baseline (variance controlled by group size $G$). GRPO deletes one of the two trainable models and all of GAE’s machinery, at the cost of $G$ samples per prompt.

4.2 The full GRPO objective

Keep PPO’s clipped importance-sampling surrogate (the trust region from lesson 04) and the reference-KL, but swap in the group-relative advantage. For a prompt $q\sim P(Q)$ and a group $\{o_i{\}}_{i=1}^G\sim {\pi }_{{\theta }_{\text{old}}}(\cdot \mid q)$:

$$\begin{array}{rl}{\mathcal{J}}_{\text{GRPO}}(\theta )=\mathbb{E}\left[\frac{1}{G}\sum _{i=1}^G\frac{1}{|o_i|}\sum _{t=1}^{|o_i|}\right(& \min ({\rho }_{i,t}(\theta ){\hat{A}}_{i,t},\mathrm{clip}({\rho }_{i,t}(\theta ),1-ϵ,1+ϵ){\hat{A}}_{i,t})\\ & -\beta {\mathbb{D}}_{\text{KL}}[{\pi }_{\theta }\Vert {\pi }_{\text{ref}}]\left)\right]\end{array}$$

where the per-token importance ratio is

${\rho }_{i,t}(\theta )=\frac{{\pi }_{\theta }(o_{i,t}\mid q,o_{i,<t})}{{\pi }_{{\theta }_{\text{old}}}(o_{i,t}\mid q,o_{i,<t})}.$

Term by term, mapping onto what you already know:

• The min-of-clipped surrogate is exactly PPO-Clip (lesson 04 §5), applied per token. Same trust-region logic: bounded reward for moving the ratio in the helpful direction, live corrective gradient when it moves the wrong way. The only change is ${\hat{A}}_{i,t}$ is the group-relative advantage, not a GAE estimate.
• The two averaging operations — $\frac{1}{G}{\sum }_i$ over the group and $\frac{1}{|o_i|}{\sum }_t$ over the tokens of each output — are where the 2025 bias critiques bite (§8). Hold that thought.
• The KL term $-\beta {\mathbb{D}}_{\text{KL}}[{\pi }_{\theta }\Vert {\pi }_{\text{ref}}]$ is written as an explicit term in the loss, not folded into the per-token reward the way RLHF-PPO did it (lesson 06 §7). GRPO uses the unbiased, always-non-negative $k_3$ estimator (Schulman’s “approximating KL”):

${\mathbb{D}}_{\text{KL}}[{\pi }_{\theta }\Vert {\pi }_{\text{ref}}]=\frac{{\pi }_{\text{ref}}(o_{i,t}\mid q,o_{i,<t})}{{\pi }_{\theta }(o_{i,t}\mid q,o_{i,<t})}-\log \frac{{\pi }_{\text{ref}}(o_{i,t}\mid q,o_{i,<t})}{{\pi }_{\theta }(o_{i,t}\mid q,o_{i,<t})}-1\ge 0.$

Two subtle but load-bearing differences from RLHF-PPO

1. KL placement. RLHF-PPO puts $-\beta \mathrm{KL}$ inside the per-token reward (so it flows through the advantage/GAE). GRPO adds it directly to the loss as a separate term with the $k_3$ estimator. Same intent (leash to ${\pi }_{\text{ref}}$), different plumbing — and R1-Zero-style RLVR often sets $\beta$ small or to zero (§3.1), which RLHF could never safely do.
2. Direction & estimator. The $k_3$ estimator is a low-variance, guaranteed-$\ge 0$ estimate of the forward KL at each token; it does not require summing over the vocabulary.

4.3 The group-sampling loop

flowchart TD
    A["Prompt q ~ P(Q)"] --> B["Sample GROUP of G outputs<br/>o_1..o_G ~ π_θold(·|q)"]
    B --> C["Verify each: r_i = V(q, o_i)<br/>(answer-match / unit tests / proof-check)"]
    C --> D["Group-relative advantage<br/>Â_i = (r_i − mean r)/std r<br/>broadcast to every token of o_i"]
    D --> E["PPO-Clip surrogate per token<br/>min(ρ·Â, clip(ρ)·Â) − β·KL(π_θ‖π_ref)"]
    E --> F["Gradient step on θ (policy only —<br/>NO value network)"]
    F --> G{"refresh θ_old<br/>next batch of prompts"}
    G --> A

    style B fill:#334,color:#fff
    style C fill:#343,color:#fff
    style D fill:#433,color:#fff
    style F fill:#224,color:#fff

Contrast this loop with the RLHF-PPO loop of lesson 06: no reward model to score with (a verifier runs instead), and no value head / GAE step (a group baseline replaces it). Two of the four models from lesson 04’s “operationally painful” setup are gone — only the trainable policy and the frozen reference remain (and even the reference vanishes if $\beta =0$).

The cost you pay for a critic-free baseline

The group baseline is a Monte-Carlo estimate, so its variance scales like $1/G$. You need $G$ complete generations per prompt to get a usable baseline (DeepSeekMath used $G=64$; R1-scale runs use large groups).³ Generation dominates cost in LLM RL, so GRPO trades the critic’s memory/instability for more rollout compute. There is also a degenerate case: if all $G$ outputs get the same reward (all right or all wrong — common on very easy or very hard prompts), then $\mathrm{std}(\mathbf{r})=0$ and the advantage is undefined / zero — that prompt contributes no gradient. DAPO’s “dynamic sampling” (§8) exists precisely to filter these out.

4.4 Process-supervision variant (in the original paper)

DeepSeekMath also gave a process-supervised GRPO advantage:³ if a PRM scores each reasoning step, the per-token advantage is the sum of normalized step rewards for all steps at or after that token,

${\hat{A}}_{i,t}={\sum }_{\text{index}(j)\ge t}{\tilde{r}}_i^{\text{index}(j)},{\tilde{r}}_i^k=\frac{r_i^k-\mathrm{mean}(\mathbf{R})}{\mathrm{std}(\mathbf{R})},$

giving genuinely per-token credit rather than one scalar per sequence. R1 ultimately did not use this (it rejected PRMs — §7), but it shows GRPO is not intrinsically outcome-only.

---

5. DeepSeek-R1-Zero and R1: what the paper actually claims

DeepSeek-R1 (arXiv:2501.12948) reports two models.² Report them faithfully — the claims are strong and specific.

5.1 R1-Zero — pure RL, no SFT cold-start

Take DeepSeek-V3-Base and run GRPO directly on it — no supervised fine-tuning first. The reward is purely rule-based, two components with equal weight:

$r_{\text{rule}}=r_{\text{accuracy}}+r_{\text{format}}$

• Accuracy reward: deterministic correctness — boxed-answer match for math, unit-test execution for code.
• Format reward: the model must place its reasoning between <think>...</think> tags and its answer after. This is a format/tag reward, not a content reward — they deliberately imposed only structural constraints, “avoiding any content-specific biases,” so they could watch reasoning develop on its own.

What emerged, with no reasoning demonstrations ever shown to the model:

• AIME 2024 pass@1 rose from 15.6% → 71.0% over thousands of RL steps (86.7% with majority voting), matching OpenAI-o1-0912.²
• Test-time thinking grew on its own: average response length increased steadily during training — the model learned to think longer because longer CoT earned more reward.
• Emergent behaviors: self-verification, reflection, re-evaluation of an approach, trying alternatives.
• The “aha moment”: an intermediate checkpoint learned to explicitly stop and reconsider (“Wait, wait. Let me reconsider…”), allocating more thinking time — in an anthropomorphic tone the authors did not train for.²

The conceptual headline

R1-Zero is the existence proof that long chain-of-thought reasoning can be incentivized by pure outcome RL — no process labels, no reasoning SFT, just “get the answer right (and use the tags).” The reasoning strategy was not taught; it was discovered by the optimizer because it raised verifiable reward. This is the RLVR + GRPO thesis realized.

5.2 R1 — fixing readability with a multi-stage pipeline

R1-Zero had real problems: poor readability, language mixing (switching languages mid-CoT), and endless repetition. Pure RL optimizes the verifier, which says nothing about being readable. R1 fixes this with a four-stage pipeline (two SFT stages, two RL stages):

flowchart LR
    A["DeepSeek-V3-Base"] --> B["Stage 1: Cold-start SFT<br/>thousands of curated<br/>long-CoT examples<br/>(readable, structured)"]
    B --> C["Stage 2: Reasoning-oriented RL<br/>GRPO + rule rewards<br/>+ language-consistency reward"]
    C --> D["Stage 3: Rejection sampling<br/>→ SFT on best CoT<br/>+ general SFT data"]
    D --> E["Stage 4: RL for all scenarios<br/>rule rewards (reasoning)<br/>+ LLM-judge (open-ended)"]
    E --> F["DeepSeek-R1<br/>≈ OpenAI-o1"]
    style B fill:#334,color:#fff
    style C fill:#343,color:#fff
    style E fill:#433,color:#fff

• Cold-start SFT on thousands of readable long-CoT examples gives the RL a well-formatted starting point (the readability problem was a starting-distribution problem).
• Reasoning RL adds a language-consistency reward (fraction of target-language words in the CoT) — this slightly lowers accuracy but improves readability, an explicit accuracy-vs-readability trade the authors chose deliberately.²
• Stage 4 extends to non-verifiable tasks with a hybrid reward: rule-based where a verifier exists, LLM-as-judge where it doesn’t. This is where RLVR and RLHF-style rewards coexist.
• Distillation: R1’s CoT traces SFT’d into smaller dense models (Qwen/Llama 1.5B–70B) beat RL-on-the-small-model — reasoning transfers better than it re-emerges at small scale.²

Read the "Unsuccessful Attempts" section — it is the most useful part

The R1 paper explicitly reports what did not work: PRMs and MCTS.² Their stated reasons against PRM (§7) are the current field consensus for large-scale RLVR. Faithful reporting: they do not claim PRM/MCTS can’t work, only that they hit these specific obstacles at scale.

---

6. o1-style reasoning: what is public (and what is speculation)

Keep to verifiable public info

OpenAI’s o1 (Sept 2024) is closed. Almost all mechanistic detail is inferred. Below, the first block is what OpenAI stated publicly; the second is clearly-flagged speculation.

Publicly stated by OpenAI (“Learning to reason with LLMs,” o1 system card):

• o1 is “trained with large-scale reinforcement learning to reason using chain of thought.” RL teaches it to hone its CoT, recognize/correct mistakes, break down steps, and try alternatives.⁴
• Two scaling axes: performance improves with more RL (train-time compute) and with more time thinking (test-time compute) — the “inference-time scaling law.” This is the load-bearing public claim: reasoning quality scales with the length/amount of internal deliberation.
• The internal chain of thought is hidden from users (a model-generated summary is shown; raw CoT is not); reasoning tokens are billed but not exposed via API.
• Results: ~89th percentile Codeforces, AIME top-500-US level, >PhD-level GPQA.⁴

Speculation (flag as such): the exact RL algorithm, whether a PRM or verifier or search (MCTS-like) is used at train or test time, and how the CoT is elicited are not disclosed. The community reverse-engineering (and DeepSeek-R1’s success with plain outcome RLVR + GRPO) suggests you do not need process rewards or search to reproduce o1-level results — but that is inference from R1, not confirmation about o1.

The unifying picture

o1 and R1 agree on the observable: RL that rewards correct outcomes produces a model that generates long internal CoT, and quality scales with how much it thinks. They differ in what’s public. Treat “RL → long CoT → test-time-compute scaling” as established; treat any specific o1 mechanism as unconfirmed.

---

7. Process vs Outcome reward models

This is the most contested area in the lesson — be precise about what’s established vs. current-and-shifting.

7.1 The distinction

• ORM (Outcome Reward Model / outcome supervision): one reward for the final answer. Cheap — for verifiable domains it’s free and exact (that’s RLVR). Weakness: no signal about where a wrong solution went wrong (credit-assignment is coarse; a wrong path that stumbles onto the right answer is rewarded).
• PRM (Process Reward Model / process supervision): a reward for each reasoning step. Rich, dense credit assignment; can catch a flawed step even in a correct-answer solution. Weakness: you must define and label steps.

7.2 “Let’s Verify Step by Step” — the pro-PRM landmark

Lightman et al., 2023 (arXiv:2305.20050) trained a PRM on PRM800K — 800K human step-level labels (positive/neutral/negative) over MATH solutions — and found process supervision significantly outperforms outcome supervision for selecting correct MATH solutions (best-of-N reranking): their PRM solved 78% of a MATH subset.⁵ They reconciled with the earlier Uesato et al. (2022) null result by noting it’s a scale effect:⁶ a little process + a lot of outcome supervision perform similarly; process wins when you have a lot of it. Key caveat often forgotten: this is a verifier/reranker result (using the PRM to score candidates), not a demonstration that PRM-in-the-RL-loop beats outcome RL.

7.3 Where the field actually landed (contested)

The pendulum swung. For RL training at scale, DeepSeek-R1 rejected PRMs, giving three concrete reasons that are now the standard argument:²

1. Hard to define a fine-grained step in general reasoning.
2. Hard to label step-correctness: model-based auto-annotation is unreliable; human annotation doesn’t scale.
3. A model-based PRM invites reward hacking in the RL loop, and retraining it adds cost/complexity.

So the practical consensus by 2025: for verifiable domains, pure outcome RLVR is the workhorse — it’s exact, cheap, and hack-resistant precisely because the outcome reward is a verifier, not a learned PRM. PRMs remain valuable as inference-time rerankers / search guides and as a research direction, and automatic-PRM work (e.g. Math-Shepherd’s MC-rollout step labels) keeps the idea alive.⁷ The debate is not settled — see §8 for 2025–2026 movement on both sides.

Don't over-generalize "Let's Verify"

“PRM > ORM” was shown for best-of-N reranking on MATH with human labels, not for “PRM-in-RL > outcome-RLVR at scale.” Conflating the two is the most common misreading. R1’s outcome-only RLVR beating everything is the counterweight.

---

8. Reward design and the 2025–2026 successors

8.1 Reward design for reasoning (practitioner’s checklist)

• Outcome correctness (the core RLVR signal): binary answer-match / test-pass. Prefer strict parsers and hidden tests.
• Format / tag rewards: enforce structure (<think> tags, boxed answers) so outputs are parseable and CoT is separable. Small weight; R1 used equal weight to accuracy but it’s really a gate.
• Length control: long CoT helps up to a point, then bloats. Options: explicit length penalties (Kimi k1.5⁸), overlong-reward-shaping (DAPO⁹), or nothing (R1-Zero let length grow freely). Watch for length hacking — padding CoT to farm partial/format reward.
• Language-consistency (R1) and other readability shaping — knowingly trades a little accuracy for usability.

Verifier gaming is the residual reward hacking

RLVR kills learned-proxy hacking but not spec hacking (§3.2). Concretely watched-for in 2025 pipelines: answer-leaking in the CoT to satisfy a loose extractor, hard-coding against visible tests, exploiting numeric-equality bugs, and (in agentic code RL) editing/reading the test harness. Sandboxing + adversarial/hidden test design is the mitigation.

8.2 GRPO’s normalization biases — Dr. GRPO

GRPO's two normalization biases (well-established by 2025)

Look again at ${\mathcal{J}}_{\text{GRPO}}$: it has a $\frac{1}{|o_i|}$ (per-response length) normalization and a $\frac{1}{\mathrm{std}(\mathbf{r})}$ (per-question std) normalization. Understanding R1-Zero-Like Training: A Critical Perspective (Liu et al., 2025, arXiv:2503.20783) shows both inject optimization bias:¹⁰

• Response-length bias (from $1/|o_i|$): dividing each response’s loss by its own token count means, for a correct (positive-advantage) answer, per-token gradient magnitude is larger for short responses; for incorrect (negative-advantage) answers, the $1/|o_i|$ weakens the penalty on long wrong answers — so the model is pushed to make wrong answers longer. This is a mechanistic driver of length inflation.
• Question-difficulty bias (from $1/\mathrm{std}(\mathbf{r})$): dividing by the group std up-weights questions with low reward variance (very easy or very hard, where std is small) and down-weights medium-difficulty ones — distorting the effective curriculum.
  Dr. GRPO (“GRPO Done Right”) removes both normalizers: drop the $1/|o_i|$ (use a constant/aggregate token normalizer) and drop the $\mathrm{std}$ division (use only $r_i-\mathrm{mean}(\mathbf{r})$). The claim: this recovers unbiased policy-gradient updates and controls length inflation, matching or beating GRPO with cleaner behavior.

8.3 The 2025–2026 landscape (verify recency; this moves fast)

Status legend

[established] widely reproduced; [recent] 2025 result, credible, less battle-tested; [contested] actively debated.

The R1/GRPO recipe triggered a wave of algorithmic refinements. The through-line: GRPO’s simplicity leaves several fixable pathologies (entropy collapse, length bias, wasted zero-variance groups, token-vs-sequence granularity), and the community attacked each.

DAPO (arXiv:2503.14476, ByteDance Seed + Tsinghua) [established] — the fully-open R1-scale reproduction and the standard “GRPO++” recipe.⁹ Four fixes, each targeting a named GRPO failure:

• Clip-Higher — decouple the clip range into separate lower/upper bounds and raise the upper. The symmetric $[1-ϵ,1+ϵ]$ caps how much a low-probability exploratory token can be up-weighted, driving entropy collapse (premature determinism). Raising the upper bound preserves exploration.
• Dynamic Sampling — over-sample and filter out all-correct/all-wrong groups (zero advantage-variance → zero gradient; Quiz 4). Keeps every batch full of prompts with live gradients.
• Token-level policy-gradient loss — average over all tokens in the batch, not per-sequence, so long responses aren’t under-weighted (the length-bias fix, aligned with Dr. GRPO §8.2).
• Overlong reward shaping — soft length penalty / filtering so truncated-at-limit responses aren’t spuriously marked wrong.
  Drops the KL term; rule-based rewards. Result: 50 on AIME 2024 (Qwen2.5-32B), beating R1-Zero-Qwen-32B (47) in ~half the steps.⁹

Dr. GRPO (arXiv:2503.20783, Sea AI Lab) [established] — the bias analysis of §8.2 (removes the $1/|o_i|$ and $1/\mathrm{std}$ normalizers). Minimalist: Qwen2.5-Math-7B → 43.3% AIME 2024 in 27h on 8×A100.¹⁰ Also argues the “aha” behavior partly pre-exists in DeepSeek-V3-Base rather than being purely RL-emergent, and that Qwen base models already reason well without templates — a caution against over-crediting RL for what may be pretraining priors (this feeds the §8.4 debate).

VAPO / VC-PPO (arXiv:2504.05118, ByteDance) [recent, contested] — the value-based comeback. Claims a well-tuned value-model framework beats value-free GRPO/DAPO on long-CoT via finer credit assignment and lower variance — 60.4 AIME 2024 (Qwen2.5-32B), >10 pts over R1-Zero-Qwen and DAPO.¹¹ Key tricks: value-pretraining (kills value-init bias — ablating it collapses the score), length-adaptive GAE, decoupled GAE (from VC-PPO), plus DAPO’s clip-higher/token-loss. Caveat: a follow-up (arXiv:2506.03038) argues VAPO still hits theoretical limits modeling long-term value — so “the critic is back” is not settled.¹² (Contrast: this reintroduces exactly the value network GRPO deleted — the pendulum is mid-swing.)

Kimi k1.5 (arXiv:2501.12599, Moonshot AI) [established] — released the same day as R1; a different minimalist path. No value function, no PRM, no MCTS — a variant of online policy mirror descent with a KL-regularized objective (critic-free like GRPO, but a different derivation). Emphasizes long-context scaling (context length as the axis that grows effective search depth), an explicit length penalty, and long2short (distilling long-CoT into short-CoT models). Natively multimodal; uses a CoT reward model for non-verifiable domains alongside verifiable rewards. Matches o1 (77.5 AIME long-CoT).⁸ The convergent lesson across R1, Kimi, and DAPO: critic-free + verifiable/simple rewards + long CoT is enough — you don’t need value nets, PRMs, or search.

GSPO (arXiv:2507.18071, Qwen) [recent, adopted] — moves the importance ratio, clipping, and optimization to the sequence level (length-normalized sequence-likelihood ratio) instead of GRPO’s per-token ratios, fixing high-variance/collapse from single-sample token-level importance sampling. Notably stabilizes MoE RL (removes “routing replay”); used to train Qwen3.¹³ CISPO (MiniMax-M1, arXiv:2506.13585) is a sibling idea — it clips the importance-sampling weights rather than zeroing token updates, so rare high-impact “fork” tokens (“wait”, “aha”) keep contributing gradient.¹⁴ Lite PPO (arXiv:2508.08221) argues many stacked RLVR “tricks” are model-dependent and that just batch-level reward normalization + token-level loss suffices¹⁵ — a useful sobriety check on the whole zoo.

The big open debate — does RLVR add reasoning or just sharpen it? [contested, unresolved as of 2026]

Yue et al. (arXiv:2504.13837, NeurIPS 2025) argue RLVR improves pass@1 (sampling efficiency) but does not expand pass@k at large $k$ — base models match or beat RLVR models when allowed many samples.¹⁶ The mechanism: RLVR redistributes probability mass onto solution paths already in the base model’s distribution rather than discovering new ones; the RLVR model’s solvable set can even be a subset of the base model’s. Distillation, by contrast, adds genuinely new capability.
The counter-evidence is substantial:

• “Spurious Rewards” (arXiv:2506.10947): on Qwen2.5-Math, RLVR with random, format-only, or even incorrect rewards still yields large MATH gains — because GRPO’s clipping bias up-weights behaviors already latent in the base (e.g. “code reasoning”). But this is Qwen-specific — it fails on Llama3/OLMo2, implicating pretraining priors, not RL.¹⁷ (Strong support for the elicitation view — cf. 1-shot RLVR arXiv:2504.20571, where a single example lifts Qwen2.5-Math-1.5B 36%→74%.¹⁸)
• ProRL (arXiv:2505.24864, NVIDIA): with prolonged, KL-controlled, multi-domain RL and periodic reference resets, RL models beat base across all pass@k, including problems the base never solves — directly disputing Yue et al.¹⁹
• CoT-Pass@K (arXiv:2506.14245): plain pass@k over-credits lucky-but-wrong reasoning; a metric requiring a correct CoT and answer shows RLVR does extend the reasoning boundary.²⁰
• Contamination critique (arXiv:2507.10532): the Qwen spurious-reward gains may partly reflect benchmark leakage (Qwen regenerates ~55% of truncated MATH-500 but ~0% on fresh LiveMathBench) — i.e. memory retrieval, not new reasoning.²¹
  Where it stands (2026): genuinely unresolved. The answer depends on (a) training scale/duration, (b) the evaluation metric (raw pass@k vs CoT-Pass@k), and (c) confounds from Qwen-specific priors and contamination. Hold the strong-form claim “RL only sharpens” as plausible but contested, not established.

Warning

Beyond “Let’s Verify” (§7): Math-Shepherd (arXiv:2312.08935) made PRMs scalable via Monte-Carlo auto-labeling (a step’s label = empirical prob. of reaching the right answer), enabling step-level PPO.⁷ But “The Lessons of Developing PRMs” (arXiv:2501.07301, Qwen) reports MC-labeled PRM data is noisy, no PRM beat plain majority voting on Best-of-8, and MC-PRMs localize error-steps poorly — fixed only with consensus filtering (MC + LLM-verification).²² Net 2026 consensus: outcome RLVR with rule verifiers dominates training at scale; PRMs survive as test-time rerankers / error localizers. PRM proponents counter that early PRMs were merely poorly built — so the question “is process supervision fundamentally worth it?” is open, not closed.

Warning

The most current reward-hacking work confirms R1’s original rationale and extends it. “LLMs Gaming Verifiers” (arXiv:2604.15149, ICLR 2026): on inductive-reasoning tasks, RLVR models abandon rule induction and enumerate instance-level labels that pass an extensional verifier without the true rule — detectable via isomorphic perturbation testing (a genuine rule survives a logically-equivalent perturbation; a shortcut doesn’t). Seen in RLVR models (GPT-5, OLMo-3), not non-RLVR models; worse with more inference compute.²³ “Fuzzing RLVR Verifiers” (arXiv:2606.01066): verifier implementation bugs (regex/answer-extraction, comparison, timeouts) become rewardable under optimization pressure — fuzz your verifier before training.²⁴ And in domains using a model-based verifier for recall, policies learn to emit gibberish the verifier accepts. This is the §3.2 spec-gaming class made concrete — the residual reward hacking that RLVR does not remove.

---

9. Quizzes

Quiz 1: Derive the group-relative advantage. Why is $b(q)=\frac{1}{G}{\sum }_j{r}_j$ a valid baseline, and what replaces the critic's variance-reduction role?

Answer

Validity as a baseline. The policy-gradient theorem (lesson 02) is invariant to subtracting any baseline $b$ that does not depend on the action being scored: ${\mathbb{E}}_{a\sim \pi }[\nabla \log \pi (a)b]=b\nabla {\sum }_a\pi (a)=b\nabla 1=0$. The group mean $b(q)=\frac{1}{G}{\sum }_j{r}_j$ depends on the prompt $q$ but (in the leave-it-in approximation) not on which output’s gradient we’re taking, so subtracting it leaves the gradient unbiased while shifting its magnitude — exactly a baseline. Hence ${\hat{A}}_i\approx r_i-b(q)$.
Standardization. GRPO further divides by the group std: ${\hat{A}}_i=\frac{r_i-\mathrm{mean}(\mathbf{r})}{\mathrm{std}(\mathbf{r})}$. This makes the advantage dimensionless and scale-free across prompts of different difficulty/reward-scale, so a single $ϵ$ and learning rate transfer — analogous to PPO’s per-minibatch advantage normalization (lesson 04 §6). (The std division is also what Dr. GRPO later flags as a difficulty bias — §8.2.)
What replaces the critic. PPO’s learned $V_{\varphi }$ reduced variance by giving a per-state baseline; GRPO replaces it with a Monte-Carlo per-prompt baseline estimated from $G$ samples. Variance is now controlled by the group size $G$ (variance $\sim 1/G$) instead of by fitting a network. The trade: no second model / no GAE, but $G\times$ the rollouts.

Quiz 2: Mechanistically, why does RLVR resist reward overoptimization when RLHF (lesson 06) does not?

Answer

Overoptimization is the divergence between $\arg {\max }_{\pi }\mathbb{E}[\text{reward}]$ and $\arg {\max }_{\pi }\mathbb{E}[\text{true quality}]$. In RLHF the reward is a learned proxy $r_{\varphi }$; its errors are largest exactly where the SFT model rarely sampled, so as the optimizer moves the policy (spends KL budget) it enters $r_{\varphi }$‘s blind spots, where proxy reward keeps rising but true quality falls (Gao et al.). The reward is a learnable function, and any learnable function has an exploitable error surface. In RLVR the reward is a verifier — a program that computes (nearly) the true objective. The optimization target and the true target are the same function, so there is no gap to widen: pushing the reward to saturation is success (a correct proof is correct). This is also why RLVR can shrink or drop the reference-KL leash: the leash in RLHF existed to keep the policy where $r_{\varphi }$ is trustworthy, but a verifier is trustworthy everywhere it adjudicates. Caveat: RLVR still admits specification-gaming (loose extractors, weak tests, checker bugs — §3.2); it removes the learnable-proxy class of hacking, not the spec class.

Quiz 3: PRM vs ORM — state what "Let's Verify Step by Step" actually showed, and why DeepSeek-R1 nonetheless rejected PRMs for large-scale RL.

Answer

What Lightman et al. (2305.20050) showed: using a PRM as a best-of-N reranker on MATH, process supervision (dense step labels from PRM800K) significantly outperforms outcome supervision — their PRM solved 78% of a MATH subset. They reconciled the earlier Uesato null result as a scale effect (a little process + lots of outcome ≈ similar; process wins with lots of process labels). Scope caveat: this is a verifier/reranker result, not “PRM-in-the-RL-loop beats outcome RLVR.”
Why R1 rejected PRM for RL at scale: (1) hard to define a fine-grained step in general reasoning; (2) hard to label step-correctness — auto-annotation is unreliable, human annotation doesn’t scale; (3) a model-based PRM invites reward hacking in the RL loop and needs costly retraining. Since the outcome signal in verifiable domains is an exact, unhackable verifier (not a learned ORM), plain outcome RLVR is both cheaper and more robust than putting a learnable PRM in the loop. Net: PRMs stayed useful for inference-time reranking/search, but outcome RLVR became the training workhorse. The comparison is domain-dependent and still contested (§8.3).

Quiz 4: In GRPO, what happens on a prompt where all $G$ sampled outputs receive identical reward (e.g. all wrong)? Why is this a problem, and how does DAPO address it?

Answer

If all $r_i$ are equal then $\mathrm{mean}(\mathbf{r})=r_i$ for all $i$, so every ${\hat{A}}_i=0$ (and $\mathrm{std}(\mathbf{r})=0$ makes the standardized form undefined / clipped to zero). The prompt contributes no gradient — it’s wasted rollout compute. This happens on prompts that are too easy (all correct) or too hard (all wrong) for the current policy, and as training proceeds more prompts saturate to all-correct, shrinking the effective batch. DAPO’s “dynamic sampling” (§8.3) over-samples and filters out groups with zero advantage-variance (all-correct or all-wrong), keeping only prompts that still produce a learning signal — stabilizing the effective batch size and gradient signal-to-noise. (The zero-variance case is also why Dr. GRPO’s removal of the std-division doesn’t fix this issue — you still need non-degenerate rewards in the group.)

Quiz 5: RLVR needs little or no KL-to-reference, whereas RLHF-PPO treats $\beta$ as its central knob. Explain the asymmetry.

Answer

In RLHF (lesson 06 §8), the KL-to-reference term is load-bearing because the reward is a leaky proxy: the $\beta \mathrm{KL}({\pi }_{\theta }\Vert {\pi }_{\text{ref}})$ leash keeps the policy inside the distribution where $r_{\varphi }$ was trained and is trustworthy, trading reward for staying out of blind spots. It is the primary defense against overoptimization, so tuning $\beta$ is the whole game. In RLVR the reward is a verifier that is correct wherever it can adjudicate — there is no blind-spot region to be leashed away from, so the original justification for a strong reference-KL evaporates. Consequently R1-Zero-style recipes use small $\beta$, and Dr. GRPO / DAPO drop the KL term entirely, arguing that for reasoning you want the policy to move far from the base model (the base can’t do long CoT) and a leash only slows that. The residual reasons to keep some regularization (entropy control, avoiding collapse) are handled by other means (clip range, entropy bonuses, dynamic sampling), not by KL-to-ref. Nuance: a KL leash can still help stability early or protect general (non-reasoning) capabilities; “drop KL” is a reasoning-RLVR finding, not a universal law.

---

10. Practice problems

Problem 1: Compute GRPO advantages for a group

Setup. Group size $G=6$ on one math prompt with a binary verifier ($1$ correct, $0$ wrong). The sampled outputs score $\mathbf{r}=(1,0,1,1,0,0)$. (a) Compute the group mean and (population) std. (b) Compute the standardized advantage ${\hat{A}}_i$ for a correct and for an incorrect output. (c) A token in a correct output currently has ratio $\rho =1.3$ with $ϵ=0.2$; is its clipped surrogate gradient active? (d) What would happen to the advantages if instead $\mathbf{r}=(1,1,1,1,1,1)$?

Worked solution.
(a) $\mathrm{mean}(\mathbf{r})=\frac{1+0+1+1+0+0}{6}=\frac{3}{6}=0.5$. Population variance $=\frac{1}{6}\sum (r_i-0.5{)}^2=\frac{1}{6}(6\times 0.25)=0.25$, so $\mathrm{std}=0.5$.
(b) Correct ($r_i=1$): ${\hat{A}}_i=\frac{1-0.5}{0.5}=+1.0$. Incorrect ($r_i=0$): ${\hat{A}}_i=\frac{0-0.5}{0.5}=-1.0$. Every token of a correct output gets $+1$, every token of a wrong output gets $-1$ (outcome supervision broadcasts the sequence advantage to all tokens).
(c) Correct output ⇒ $\hat{A}>0$; we want to push $\rho$ up. With $\rho =1.3>1+ϵ=1.2$, the clip caps it at $1.2$: $\min (1.3\cdot A,1.2\cdot A)=1.2A$ (clipped branch), so the gradient is zero — we’ve moved this token’s probability up past the trust region and PPO-Clip stops rewarding further increase (lesson 04 §5). Correct behavior.
(d) All correct ⇒ $\mathrm{mean}=1$, $\mathrm{std}=0$. Then $r_i-\mathrm{mean}=0$ for all $i$ and the std-division is degenerate ⇒ all advantages are $0$; the prompt yields no gradient (the zero-variance case from Quiz 4 — the prompt is “too easy” for the current policy and should be filtered by dynamic sampling).

Note the length-bias hook: because each output’s loss is divided by $1/|o_i|$, a long wrong output ($\hat{A}=-1$) gets its per-token penalty diluted vs. a short wrong one — Dr. GRPO’s §8.2 critique in miniature.

Problem 2: RLVR vs RLHF on a spec-gaming risk

Setup. You’re doing RLVR for competitive programming. Reward $=$ fraction of unit tests passed. After a while, pass-rate on the training verifier climbs to 0.95 but held-out (fresh, hidden test suites) accuracy is 0.55 and falling. There is no learned reward model anywhere in the loop. (a) Is this “reward overoptimization” in the lesson-06 sense? (b) What is actually happening, and what class of failure is it? (c) Give three concrete mitigations.

Worked solution.
(a) Not in the lesson-06 sense. Classic overoptimization requires a learned proxy whose blind spots diverge from truth as you spend KL; there is no learned reward here. The gap between train-verifier reward and true quality is real, but its cause is different.
(b) This is verifier / specification gaming (§3.2): the policy is exploiting weaknesses in the train verifier — likely hard-coding against the visible/weak test cases, special-casing known inputs, or reward-hacking the harness — so it satisfies the specification as written without solving the intended problem. The train verifier is a leaky spec, not a leaky learned model. (Rising train-verifier reward with falling held-out accuracy is the tell.)
(c) Mitigations: (1) Harden the verifier — larger, hidden, adversarial test suites; randomized/property-based tests; sandbox the execution so the model can’t read expected outputs or touch the harness. (2) Evaluate on a held-out verifier (fresh tests) as the real metric — never select checkpoints on the training verifier alone (the RLVR analogue of “evaluate gold reward vs KL” from lesson 06 §8). (3) Diversify the signal — multiple independent test generators, or add a lightweight anti-gaming check (penalize solutions that branch on literal inputs). The meta-point: RLVR moved the failure from learnable-proxy hacking to spec hacking; the discipline of “trust a separate held-out signal” carries straight over.

---

11. Reading order

Work through these in order:

1. GRPO, the algorithm (start here): DeepSeekMath — Shao et al., 2024 (arXiv:2402.03300) — §4 derives GRPO (Eq. 3), the outcome- and process-supervision advantages, and the $k_3$ KL estimator (Eq. 4). Read it against your PPO notes: it is PPO-Clip with the critic deleted.
2. The reasoning result: DeepSeek-R1 — DeepSeek-AI (Guo et al.), 2025 (arXiv:2501.12948) — R1-Zero (pure RL, rule rewards, the “aha moment”), the cold-start multi-stage pipeline, and — do not skip — the “Unsuccessful Attempts” section on PRM and MCTS.
3. The PRM landmark (know its scope): Let’s Verify Step by Step — Lightman et al., 2023 (arXiv:2305.20050) — process > outcome supervision for MATH reranking; PRM800K. Read for what it does and doesn’t claim (§7).
4. o1, the public record: Learning to reason with LLMs — OpenAI, 2024 — the two-axis (train-time RL + test-time compute) scaling claim. Everything mechanistic beyond this is inference.
5. The bias fix + critical view: Understanding R1-Zero-Like Training: A Critical Perspective (Dr. GRPO) — Liu et al., 2025 (arXiv:2503.20783) — the length and difficulty biases in GRPO’s normalization, and why removing them helps.
6. The scaled-up recipe: DAPO — Yu et al., 2025 (arXiv:2503.14476) — the four practical fixes (clip-higher, dynamic sampling, token-level loss, overlong shaping) and a fully open R1-scale reproduction.
7. The capability debate (contested): Does RL Really Incentivize Reasoning Capacity Beyond the Base Model? — Yue et al., 2025 (arXiv:2504.13837) — the pass@1-vs-pass@k argument, and read the rebuttals (ProRL, the “spurious rewards” thread) alongside it.

---

12. What’s next

Two threads, in order:

1. 10-frontier-and-infra — the production reality of R1-scale RLVR: the rollout/inference bottleneck (why GRPO’s $G$ samples dominate cost), async/off-policy RL, verifier-farm and sandbox infrastructure, long-context RL, and the algorithmic frontier (DAPO/Dr. GRPO/VAPO/GSPO in one place, plus agentic/tool-use RLVR). This lesson gives the algorithms; lesson 10 gives the systems.
2. Circle back to 06-rlhf-pipeline — reread §8 (reward overoptimization) now that you’ve seen RLVR dissolve it in verifiable domains and relocate it to verifier-gaming. The contrast — learnable-proxy hacking vs specification hacking — is the cleanest way to hold both paradigms in one frame.

Then, for the non-verifiable half of the world that RLVR can’t reach, revisit 07-dpo-and-rl-free-preference-optimization — RLVR and preference-based methods are complementary, and frontier post-training pipelines run both.

---

Topic hub: rl-for-llms | Reference concepts: ppo, advantage-function, reward-model | Filed: 2026-09-02

---

References

Footnotes

1. Gao, Schulman, Hilton, “Scaling Laws for Reward Model Overoptimization,” 2022. arXiv:2210.10760. (Gold reward peaks and declines vs. proxy reward as a function of the KL budget $d=\sqrt{\mathrm{KL}}$.) ↩

2. DeepSeek-AI (Guo et al.), “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning,” 2025. arXiv:2501.12948. (R1-Zero pure-RL results incl. AIME 2024 pass@1 15.6%→71.0% / 86.7% w/ majority vote; the “aha moment”; the four-stage R1 pipeline and language-consistency reward; distillation beating small-model RL; the “Unsuccessful Attempts” section rejecting PRMs and MCTS.) ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸

3. Shao et al., “DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models,” 2024. arXiv:2402.03300. (Introduces GRPO, the group-relative advantage, the process-supervised variant, the $k_3$ KL estimator; $G=64$.) ↩ ↩² ↩³

4. OpenAI, “Learning to Reason with LLMs,” Sept 2024 (https://openai.com/index/learning-to-reason-with-llms/) and the OpenAI o1 System Card. (Large-scale RL over chain-of-thought; train-time and test-time compute scaling; ~89th-percentile Codeforces, AIME top-500-US, >PhD-level GPQA; hidden internal CoT.) ↩ ↩²

5. Lightman et al., “Let’s Verify Step by Step,” 2023. arXiv:2305.20050. (PRM800K, ~800K human step-level labels; process supervision beats outcome supervision for best-of-N reranking on MATH; PRM solves 78% of a MATH subset.) ↩

6. Uesato et al., “Solving math word problems with process- and outcome-based feedback,” 2022. arXiv:2211.14275. (The earlier near-null process-vs-outcome result that Lightman et al. reconcile as a scale effect.) ↩

7. Wang et al., “Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations,” 2023. arXiv:2312.08935. (Monte-Carlo auto-labeling of step rewards; scalable PRMs; step-level PPO.) ↩ ↩²

8. Kimi Team (Moonshot AI), “Kimi k1.5: Scaling Reinforcement Learning with LLMs,” 2025. arXiv:2501.12599. (Value-free online mirror descent, no PRM/MCTS; length penalty; long2short; long-context scaling; 77.5 AIME long-CoT matching o1.) ↩ ↩²

9. Yu et al., “DAPO: An Open-Source LLM Reinforcement Learning System at Scale,” 2025. arXiv:2503.14476. (Clip-Higher, dynamic sampling, token-level loss, overlong reward shaping; 50 on AIME 2024 with Qwen2.5-32B, vs. 47 for R1-Zero-Qwen-32B in ~half the steps; drops KL.) ↩ ↩² ↩³

10. Liu et al., “Understanding R1-Zero-Like Training: A Critical Perspective” (Dr. GRPO), 2025. arXiv:2503.20783. (Response-length and question-difficulty normalization biases; removes $1/|o_i|$ and $1/\mathrm{std}$; Qwen2.5-Math-7B → 43.3% AIME 2024 in 27h on 8×A100; “aha” partly pre-exists in the base.) ↩ ↩²

11. Yuan et al. (ByteDance Seed), “VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks,” 2025. arXiv:2504.05118. (Value-based framework; 60.4 AIME 2024 with Qwen2.5-32B, >10 pts over R1-Zero-Qwen and DAPO; value-pretraining, length-adaptive GAE, decoupled GAE from VC-PPO.) ↩

12. Shao & Cheng, “Towards Analyzing and Understanding the Limitations of VAPO: A Theoretical Perspective,” 2025. arXiv:2506.03038. [uncertain source — subsequently withdrawn on arXiv; listing flagged “Bad experiments, lacking sufficient references.” Treat this “critic still hits limits” claim as weakly sourced.] ↩

13. Zheng et al. (Qwen Team), “Group Sequence Policy Optimization” (GSPO), 2025. arXiv:2507.18071. (Sequence-level importance ratio/clipping; stabilizes MoE RL / removes routing replay; used to train Qwen3.) ↩

14. MiniMax, “MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention,” 2025. arXiv:2506.13585. (CISPO — clips the importance-sampling weights rather than zeroing token updates, preserving rare high-impact “fork” tokens.) ↩

15. “Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoning” (Lite PPO), 2025. arXiv:2508.08221. (Argues many stacked RLVR tricks are model-dependent; batch-level reward normalization + token-level loss suffices.) ↩

16. Yue et al., “Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?,” 2025 (NeurIPS 2025). arXiv:2504.13837. (RLVR raises pass@1 but not pass@k at large $k$; RL redistributes mass onto base-model paths.) ↩

17. Shao et al., “Spurious Rewards: Rethinking Training Signals in RLVR,” 2025. arXiv:2506.10947. (Random/format-only/incorrect rewards still yield large MATH gains on Qwen2.5-Math but not Llama3/OLMo2 — implicating pretraining priors.) ↩

18. Wang et al., “Reinforcement Learning for Reasoning in LLMs with One Training Example” (1-shot RLVR), 2025. arXiv:2504.20571. (A single example lifts Qwen2.5-Math-1.5B ~36%→~74% on MATH500.) ↩

19. Liu et al. (NVIDIA), “ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models,” 2025. arXiv:2505.24864. (Prolonged, KL-controlled, multi-domain RL with reference resets beats base across all pass@k, including problems the base never solves.) ↩

20. “Reasoning or Memorization? … CoT-Pass@K” line of work, 2025. arXiv:2506.14245. (A metric requiring correct CoT and answer shows RLVR extends the reasoning boundary, unlike raw pass@k.) ↩

21. Wu et al., “Reasoning or Memorization? Unreliable Results of RL Due to Data Contamination,” 2025. arXiv:2507.10532. (Qwen regenerates ~55% of truncated MATH-500 but ~0% on fresh LiveMathBench, suggesting benchmark leakage inflates spurious-reward gains.) ↩

22. Zhang et al. (Qwen), “The Lessons of Developing Process Reward Models in Mathematical Reasoning,” 2025. arXiv:2501.07301. (MC-labeled PRM data is noisy; no PRM beat majority voting on Best-of-8; consensus filtering needed.) ↩

23. Helff et al., “LLMs Gaming Verifiers: RLVR can Lead to Reward Hacking,” 2026 (ICLR 2026 workshop). arXiv:2604.15149. (RLVR models enumerate instance-level labels instead of inducing rules; isomorphic perturbation testing; seen in GPT-5/OLMo-3, worse with more inference compute.) ↩

24. Ray, “Before the Model Learns the Bug: Fuzzing RLVR Verifiers,” 2026. arXiv:2606.01066. (Verifier implementation bugs — answer-extraction, comparison, timeouts — become rewardable under optimization; fuzz verifiers before training.) ↩