RLHF (Reinforcement Learning from Human Feedback)

Definition

Reinforcement Learning from Human Feedback (RLHF) is the post-training pipeline that aligns an LLM to human preferences by (1) collecting preference comparisons, (2) fitting a reward model, and (3) optimizing the LLM policy against that reward with a policy-gradient method — classically PPO — under a KL leash to a frozen reference. It is the applied instance of the entire RL stack: the LLM is the policy, the token sequence is a degenerate MDP, and generating a completion is a trajectory rollout.

How it works

Three stages (InstructGPT, Ouyang et al. 2022; arXiv:2203.02155):

1. SFT — supervised fine-tune on demonstrations → the reference/init policy ${\pi }_{\text{ref}}$.
2. Reward modeling — fit $r_{\psi }$ on pairwise human comparisons (Bradley–Terry).
3. RL optimization — maximize expected reward with a reference-KL penalty:
   ${\max }_{\theta }{\mathbb{E}}_{x\sim \mathcal{D},y\sim {\pi }_{\theta }}[r_{\psi }(x,y)-\beta \mathrm{KL}({\pi }_{\theta }(\cdot \mid x)\Vert {\pi }_{\text{ref}}(\cdot \mid x)\left)\right]$

Optimized token-by-token: reward is the terminal RM score minus a per-token KL, converted to per-token advantages by GAE and fed to the clipped surrogate. At its core this is REINFORCE — advantage-weighted MLE on completions — wrapped in a trust region. RLHF-PPO keeps four models in play (policy, critic/value head, frozen reference, reward model).

Why it matters

RLHF is how frontier chat models are made helpful, harmless, and instruction-following — the step where a model optimizes for something you can score but not demonstrate token-by-token. Its central risk is reward hacking: the RM is an imperfect proxy, so the KL leash and $\beta$ tuning are load-bearing. Its operational weight and instability motivate the modern successors: GRPO/RLOO (drop the critic, use a group baseline), DPO (drop the RL loop — the reference-KL objective has a closed-form optimum, fit directly on preference pairs), RLAIF/Constitutional AI (AI feedback), and RLVR (verifiable rule rewards for reasoning).

Taught in

• 04-ppo — RLHF as PPO applied to language generation: the four-model setup, the reward, and the two KLs.
• 01-mdps-and-the-rl-objective, 02-policy-gradients — the foundations RLHF instantiates.

See also

• reward-model — the learned preference proxy
• kl-regularization-rlhf — the reference-KL leash
• ppo — the classic RLHF optimizer
• policy-gradient-theorem — the gradient underneath
• advantage-function — per-token credit assignment
• rl-for-llms — the broader curriculum