Reward Model

Definition

A reward model (RM) is a learned scalar-valued function $r_{\psi }(x,y)$ that scores a completion $y$ to prompt $x$, trained to approximate human preference so it can supply the reward signal for RLHF optimization. It replaces the environment reward of a classical MDP: because human preference cannot be demonstrated token-by-token but can be scored, the RM is the “environment” that PPO (or REINFORCE/GRPO) optimizes against. It is typically an LLM with a scalar head, initialized from the SFT model.

Key math

Trained from pairwise comparisons $(x,y_w,y_l)$ (winner $\succ$ loser) under the Bradley–Terry model, minimizing:
$L(\psi )=-{\mathbb{E}}_{(x,y_w,y_l)}\left[\log \sigma \right(r_{\psi }(x,y_w)-r_{\psi }(x,y_l)\left)\right]$

At inference the RM outputs a single scalar for a whole response, available only at the final token. In RLHF-PPO this becomes the terminal reward, combined with a per-token KL penalty:
$r_t=r_{\psi }(x,y_{1:T})1[t=T]-\beta (\log {\pi }_{\theta }(y_t\mid x,y_{<t})-\log {\pi }_{\text{ref}}(y_t\mid x,y_{<t}))$

The GAE backward pass then distributes this sequence-level scalar into per-token advantages.

Why it matters

The RM is a learned, imperfect proxy for the true objective, and this is the crux of RLHF’s difficulty: its errors are largest in output regions the SFT model rarely visits, so an unconstrained optimizer will exploit them — reward hacking / over-optimization, producing text that scores high but is degenerate or dishonest. This is exactly why the reference-KL penalty is load-bearing. RM quality, calibration, and robustness bound the entire pipeline; alternatives include RLAIF (AI feedback), rule/verifier rewards (RLVR), and DPO (which folds the implicit reward into the policy loss and skips an explicit RM).

Taught in

• 04-ppo — the RM as the sparse terminal reward in RLHF-PPO and the source of over-optimization the KL guards against.

See also

• rlhf — the pipeline the RM feeds
• kl-regularization-rlhf — the leash against RM exploitation
• ppo — the optimizer that consumes RM scores
• advantage-function — how the scalar RM score becomes per-token signal
• rl-for-llms