Reward Overoptimization (Goodhart)

Definition

Reward overoptimization is the defining failure of RLHF: you care about a gold objective $R^{\ast }$ (true human preference) but can only optimize a proxy $r_{\varphi }$ fit to finite, noisy data. The two agree on the training distribution and diverge off it. The RL optimizer’s job is to find high-$r_{\varphi }$ inputs — and the largest positive errors $r_{\varphi }-R^{\ast }$ live exactly where the reward model saw little data (far from ${\pi }_{\text{ref}}$). So optimization pressure and proxy-error are positively coupled: proxy reward rises monotonically while gold reward rises, peaks, then falls. This is Goodhart’s law made mechanical: “when a measure becomes a target, it ceases to be a good measure.”

Key math — the Gao et al. scaling law

Gao, Schulman & Hilton (2022; arXiv:2210.10760) made it predictable by measuring optimization pressure not in steps but as distance from the reference in KL:
$d=\sqrt{\mathrm{KL}(\pi \Vert {\pi }_{\text{ref}})}.$
Fitting gold reward (via a much larger held-out gold RM) as a function of $d$ gives clean, humped forms — a rising term times a term that decays in $d$:
$R_{\text{gold}}^{\text{BoN}}(d)=d({\alpha }_{\text{bon}}-{\beta }_{\text{bon}}d),R_{\text{gold}}^{\text{RL}}(d)=d({\alpha }_{\text{RL}}-{\beta }_{\text{RL}}\log d).$
Findings: larger RMs overoptimize less (smaller $\beta$, higher/later peak); more RM data raises the whole curve; policy size barely affects the shape (overopt is a property of the RM). Best-of-N’s KL is analytic (${\mathrm{KL}}_{\text{BoN}}\approx \log n-\frac{n-1}{n}$), so BoN and RL sit on the same $d$-axis. The $d=\sqrt{\mathrm{KL}}$ ruler turns the KL budget from a mysterious knob into the principled x-axis of the reward-vs-quality trade-off.

The mitigation toolkit

Every mitigation either makes the proxy harder to exploit or doesn’t push as hard:

1. KL budget ($\beta$). The primary knob — $\beta \mathrm{KL}({\pi }_{\theta }\Vert {\pi }_{\text{ref}})$ keeps the policy in the RM’s competence region; sit near the gold peak. See kl-regularization-rlhf.
2. RM ensembles. Disagreement across $M$ RMs is a free OOD detector; be conservative (reward = mean − λ·std, or min).
3. WARM (Ramé et al., arXiv:2401.12187) — average the weights of $M$ RMs fine-tuned from a shared init (linearly mode-connected). Ensemble-grade robustness at one forward pass; more robust under distribution shift.
4. Uncertainty penalty — subtract an uncertainty estimate so the optimizer is repelled from unsure regions (optimize a reward lower bound).
5. Iterative RM retraining — the root-cause fix: run in rounds, collect fresh comparisons on the current policy’s outputs, retrain the RM so its competence region tracks the policy.

Why it matters

Rising RM score is not success — it is the expected behavior of an adversary against a leaky proxy. Concrete hacks: length hacking, sycophancy, formatting/markdown spam, degenerate token spikes. Always evaluate a held-out gold signal as a function of KL, expect a hump, and stop near its peak. Overoptimization is invariant to where the proxy came from — human, AI (RLAIF/CAI Goodhart too). The paradigm shift that largely dissolves it is RLVR — a verifier is not a learnable function, so there is no error surface to slip on (though spec-gaming remains). At the lab level, the benchmark itself becomes the thing Goodharted.

Taught in

• 08-scaling-rlhf-and-alternatives — §6 Goodhart made mechanical, the Gao scaling law, the full toolkit.
• 09-rl-for-reasoning — why RLVR sidesteps it; 10-frontier-and-infra — monitoring at scale, benchmark overfitting.

See also

• reward-model — the learnable proxy that leaks
• kl-regularization-rlhf — the primary leash against it
• rlvr — the paradigm that dissolves it in verifiable domains
• best-of-n-sampling — the other procedure on the $d=\sqrt{\mathrm{KL}}$ axis
• rlaif — AI feedback overoptimizes too
• rl-for-llms