Chain-of-Thought RL (emergent reasoning)

Definition

Chain-of-thought RL is the finding that long, self-correcting chain-of-thought (CoT) reasoning can be incentivized by pure outcome RL — no process labels, no reasoning demonstrations, just “get the answer right.” Running GRPO with verifiable rewards on a base model, the optimizer discovers extended reasoning (self-verification, reflection, trying alternatives) because longer, more careful CoT earns more verifiable reward. Reasoning quality then scales with test-time compute — how much the model thinks.

How it works

DeepSeek-R1-Zero (arXiv:2501.12948) is the existence proof: take DeepSeek-V3-Base, run GRPO directly (no SFT cold-start), with a purely rule-based reward
$r_{\text{rule}}=r_{\text{accuracy}}+r_{\text{format}}$
— correctness (boxed-answer match / unit tests) plus a format reward requiring reasoning inside <think>...</think> tags (deliberately no content-specific bias). What emerged with zero reasoning demonstrations:

• AIME 2024 pass@1: 15.6% → 71.0% (86.7% w/ majority voting), matching o1-0912.
• Test-time thinking grew on its own — mean response length rose steadily during training.
• Emergent self-verification, reflection, re-evaluation; the “aha moment” (an intermediate checkpoint learning to stop and reconsider).

R1-Zero had poor readability / language-mixing, fixed by R1’s four-stage pipeline (cold-start SFT on readable long-CoT → reasoning RL + language-consistency reward → rejection-sampling SFT → RL for all scenarios with a hybrid verifier + LLM-judge reward). o1 (OpenAI, Sept 2024, closed): publicly stated it is “trained with large-scale RL to reason using chain of thought,” with two scaling axes — more RL (train-time) and more thinking (test-time), the inference-time scaling law; the raw CoT is hidden. Everything mechanistic beyond this is inference; R1’s success suggests you do not need PRMs or search to reach o1-level results.

Why it matters

This is the pivot from “RL polishes a policy” to “RL discovers a capability.” The reasoning strategy was not taught — it was found by the optimizer because it raised verifiable reward, and it transfers: R1’s CoT traces SFT’d into small dense models beat RL run directly on the small model (reasoning distills better than it re-emerges at small scale). The big open debate: does RLVR create new reasoning or merely elicit/sharpen what pretraining already put there? Yue et al. (arXiv:2504.13837) argue RL raises pass@1 but not pass@k (elicitation); ProRL (arXiv:2505.24864) counters that prolonged RL expands the boundary. Unresolved as of 2026, and it decides whether the next capability jump comes from pretraining or RL FLOPs.

Taught in

• 09-rl-for-reasoning — R1-Zero/R1, the “aha moment,” o1’s public record, the create-vs-elicit debate.
• 10-frontier-and-infra — entropy collapse as the pathology that caps long-CoT exploration.

See also

• rlvr — the verifiable reward that incentivizes CoT
• grpo — the critic-free optimizer used
• process-reward-model — the step-level alternative R1 rejected
• agentic-rl — multi-turn extension of long-horizon reasoning
• rl-post-training-infra — the systems that make long-CoT RL run
• rl-for-llms