RLVR (RL with Verifiable Rewards)

Definition

RL with Verifiable Rewards replaces the learned reward model $r_{\varphi }$ with a deterministic verifier $V(x,y)$ — a program you write, not a network you fit — in domains with programmatically checkable answers (math, code, formal proofs, format). The reward is the ground-truth objective (up to verifier coverage), so there is no proxy to overfit. This largely dissolves reward overoptimization, the defining failure of RLHF. It is one of two independent deletions behind DeepSeek-R1: RLVR removes the learned RM, GRPO removes the value net.

How it works

The verifier returns a reward, typically binary or a small discrete set:

Domain	Verifier	Reward
Math (final-answer)	parse boxed answer, symbolic/numeric equality to key	$1$/$0$
Code	compile + run hidden unit tests	fraction passed, or all-or-nothing
Formal proofs	Lean/Coq/Isabelle proof checker	$1$ if accepted
Format	regex/parser on required tags	$1$ if well-formed

Why it resists overoptimization: in RLHF, $\arg {\max }_{\pi }\mathbb{E}[r_{\varphi }]$ and $\arg {\max }_{\pi }\mathbb{E}[\text{true quality}]$ diverge once you leave the RM’s competence region — that divergence is overoptimization (Gao et al.). With a verifier the optimization target and the true target are the same function, so pushing the reward to saturation is success. A consequence: RLVR needs a much smaller or zero reference-KL leash — the leash existed to protect a proxy that no longer exists, so Dr. GRPO and DAPO drop the KL term entirely.

The limits — spec-gaming, not a free lunch

RLVR removes the learnable-proxy class of reward hacking but not the specification-gaming class:

• Answer-extraction exploits — loose regex accepts the key anywhere; the model emits the answer without valid reasoning.
• Test-suite gaming — hard-coding against visible tests (if input==known: return known); reward-hacking the harness (exit 0, read expected output, monkey-patch the framework).
• Checker false positives — numeric-equality bugs ($\frac{1}{2}\ne 0.5$), rewarding right answers from wrong reasoning (outcome-only credit), sorry/admit escape hatches in proofs.
• Coverage — open-ended writing, dialogue, helpfulness, honesty have no verifier; RLVR complements rather than replaces RLHF/DPO there.

Recent work confirms even rule verifiers get gamed (“Fuzzing RLVR Verifiers,” arXiv:2606.01066; “LLMs Gaming Verifiers,” arXiv:2604.15149). Mitigation: harden verifiers (hidden/adversarial tests, strict parsers, sandboxing) and evaluate on a held-out verifier.

Why it matters

RLVR is the reasoning engine of the 2025–26 frontier stack: verifiable rewards + a GRPO-family algorithm is the workhorse for math/code/reasoning, precisely because correctness is not a matter of taste and cannot be Goodharted the way a learned RM can. It relocates reward hacking from learnable-proxy to spec hacking — the cleanest way to hold both paradigms in one frame. Note RFT/ReST already used verifiable rewards in their filter step; RLVR is that signal run through online RL instead of offline SFT.

Taught in

• 09-rl-for-reasoning — the verifier-as-reward move, why it resists overoptimization, spec-gaming limits.
• 10-frontier-and-infra — verifier/sandbox infrastructure as a first-class throughput and attack-surface concern.

See also

• grpo — the usual critic-free optimizer for RLVR
• reward-overoptimization — the failure RLVR dissolves (and relocates)
• chain-of-thought-rl — what outcome RLVR produces
• process-reward-model — the step-level alternative R1 rejected
• reward-model — the learned proxy RLVR replaces
• rl-post-training-infra — verifier farms and sandboxing
• rl-for-llms