Process Reward Model (PRM)

Definition

A Process Reward Model (PRM) supplies a reward for each reasoning step, in contrast to an Outcome Reward Model (ORM), which scores only the final answer. PRMs give dense, step-level credit assignment — they can flag a flawed step even inside a correct-answer solution — at the cost of having to define and label steps. In verifiable domains the ORM is free and exact (that is RLVR); the PRM’s promise is richer supervision where the outcome signal is too sparse.

How it works

• ORM / outcome supervision: one reward for the whole trajectory. Cheap, groundable, but coarse — a wrong path that stumbles onto the right answer is rewarded.
• PRM / process supervision: a reward $r_i^k$ per step $k$. In a process-supervised GRPO variant, the per-token advantage sums normalized step rewards from that token onward: ${\hat{A}}_{i,t}={\sum }_{\text{index}(j)\ge t}{\tilde{r}}_i^{\text{index}(j)}$ — genuine per-token credit rather than one broadcast scalar.

Labeling PRMs is the hard part:

• Human step labels — “Let’s Verify Step by Step” (Lightman et al. 2023, arXiv:2305.20050) trained a PRM on PRM800K (800K human step labels over MATH) and found process supervision significantly beats outcome supervision for best-of-N reranking (solved 78% of a MATH subset). Reconciled the earlier Uesato et al. (2022) null result as a scale effect. Scope caveat: this is a reranker/verifier result, not “PRM-in-the-RL-loop beats outcome RLVR.”
• Automatic labels — Math-Shepherd (arXiv:2312.08935) makes PRMs scalable via Monte-Carlo auto-labeling: a step’s label = empirical probability that continuing from it reaches the right answer, enabling step-level PPO without human annotation.

Why it matters

The PRM-vs-ORM debate is the most contested area of reasoning RL. DeepSeek-R1 rejected PRMs for large-scale RL on three grounds now standard: (1) hard to define a fine-grained step in general reasoning; (2) hard to label step-correctness (auto-annotation noisy, human doesn’t scale); (3) a model-based PRM invites reward hacking in the RL loop and needs costly retraining. So the 2026 consensus: for verifiable domains, outcome RLVR with rule verifiers dominates training at scale (the outcome signal is an exact verifier, not a learnable ORM), while PRMs survive as inference-time rerankers / error localizers. Counter-evidence keeps the question open: “The Lessons of Developing PRMs” (Qwen, arXiv:2501.07301) found MC-labeled PRM data noisy and no PRM beating majority voting on Best-of-8 — but proponents argue those PRMs were merely poorly built. Do not over-generalize “Let’s Verify”: “PRM > ORM” was shown for reranking with human labels, not for RL at scale.

Taught in

• 09-rl-for-reasoning — PRM vs ORM, “Let’s Verify,” Math-Shepherd, why R1 rejected PRMs.
• 10-frontier-and-infra — process vs outcome credit assignment over long CoT as an open problem.

See also

• rlvr — the outcome-verifier workhorse PRMs compete with
• reward-model — the outcome-level learned scorer (ORM)
• chain-of-thought-rl — the long reasoning traces being supervised
• best-of-n-sampling — where PRM reranking shines
• grpo — the algorithm with a process-supervised variant
• rl-for-llms