06 · The RLHF Pipeline

What you're learning

How you get from a pile of human preference judgments to a policy you can actually deploy — the full InstructGPT recipe: SFT → reward model → PPO against the RM with KL-to-reference. By the end you should be able to derive the Bradley-Terry reward loss from first principles, explain exactly why the pipeline is built in three stages instead of one, and reason mechanistically about its dominant failure mode — reward overoptimization / Goodhart.

This is Tier 2, lesson 2. It assumes you have the PPO machinery from 04-ppo and the token-level MDP + KL-to-reference plumbing from 05-rl-on-token-sequences. Lesson 04 already told you how PPO consumes a reward and a frozen reference; this lesson is about where that reward comes from and how the whole loop is assembled and where it breaks.

---

1. Learning map

graph TD
    A["The alignment problem:<br/>'be helpful/honest/harmless'<br/>has no differentiable loss"] --> B["Can't write reward by hand<br/>→ learn it from humans"]
    B --> C["Absolute scores are noisy<br/>→ collect comparisons instead"]
    C --> D["Bradley-Terry model<br/>P(y_w≻y_l)=σ(r_w−r_l)"]
    D --> E["Reward model r_φ<br/>scalar head on a transformer"]
    E --> F["RM loss<br/>−E[log σ(r_w−r_l)]"]
    F --> G["Reward defined only<br/>up to additive constant<br/>→ mean-center"]
    A --> H["Stage 1: SFT<br/>(demonstrations)"]
    H --> E
    H --> I["π_ref = frozen SFT model"]
    E --> J["Stage 3: PPO<br/>maximize r_φ − β·KL(π‖π_ref)"]
    G --> J
    I --> J
    J --> K["Failure: reward<br/>overoptimization / Goodhart<br/>(Gao et al. scaling laws)"]
    K --> L["Control knob: KL budget β"]
    J --> M["Next: DPO (07) collapses 2+3<br/>RLAIF/CAI (08) swaps human→AI<br/>RLVR/GRPO (09) swaps RM→verifier"]

    style D fill:#44a,color:#fff
    style J fill:#4a4,color:#fff
    style K fill:#a44,color:#fff

Prerequisites (assumed): the PPO clipped surrogate and the four-model RLHF-PPO setup (04-ppo); the autoregressive-LM-as-MDP framing, per-token reward, and the $\beta \mathrm{KL}({\pi }_{\theta }\Vert {\pi }_{\text{ref}})$ term (05-rl-on-token-sequences).

---

2. Why this matters: there is no loss function for “good”

Supervised fine-tuning maximizes $\log {\pi }_{\theta }(y\mid x)$ on curated demonstrations. That teaches format and behavior imitation, but it has a hard ceiling: it can only ever reproduce the demonstration distribution, and it optimizes likelihood of one reference answer, not quality of the model’s own samples. Two responses can be equally likely under the demonstrations yet wildly different in helpfulness, and SFT has no way to express “this sample is better than that one.”

The properties we actually want — helpful, honest, harmless, well-calibrated, non-sycophantic — have no closed-form differentiable loss. You cannot write loss = -helpfulness(y). What you can do is show two candidate responses to a human and ask “which is better?” That is a cheap, reliable signal. RLHF is the machinery that turns a stream of those pairwise judgments into a scalar reward, and then into gradient steps on the policy.¹²

The core move

RLHF factorizes an impossible problem into three tractable ones: (1) imitate good behavior to get a sane starting policy (SFT), (2) distill human judgment into a differentiable scalar $r_{\varphi }$ (reward modeling), (3) optimize the policy against that scalar with a leash back to the starting point (PPO + KL). Each stage exists because the previous one leaves a specific gap.

Why not learn the reward and optimize it end-to-end in one loop? Because the reward is a learned, imperfect proxy, and an RL optimizer is an adversary that will find every crack in it. Decoupling RM training from policy optimization lets you (a) train the RM to convergence on a fixed, curated preference set before exposing it to an adversary, and (b) control how hard you push against it via the KL budget. Hold that thought — §7 is entirely about what happens when the proxy leaks.

---

3. The three-stage recipe (end-to-end first)

Here is the whole InstructGPT pipeline before we drill into any single box.³

flowchart TD
    subgraph S1["Stage 1 — SFT"]
        P0["Pretrained base LM"] --> SFT["Fine-tune on human<br/>demonstrations (x, y*)<br/>maximize log π(y*|x)"]
        SFT --> M1["π_SFT"]
    end
    M1 --> REF["Freeze copy → π_ref"]
    M1 --> INIT1["Init RM backbone"]
    M1 --> INIT2["Init policy π_θ"]

    subgraph S2["Stage 2 — Reward Model"]
        INIT1 --> RMB["Transformer + scalar head r_φ(x,y)"]
        PROMPTS["Prompts x"] --> GEN["Sample K completions per prompt<br/>from an SFT-class model"]
        GEN --> RANK["Humans rank / compare pairs<br/>→ (x, y_w, y_l)"]
        RANK --> RMLOSS["Train r_φ:<br/>−E[log σ(r_φ(x,y_w)−r_φ(x,y_l))]"]
        RMB --> RMLOSS
        RMLOSS --> M2["r_φ (frozen after training)"]
    end

    subgraph S3["Stage 3 — PPO"]
        INIT2 --> POL["Policy π_θ (+ value head)"]
        POL --> ROLL["Sample y ~ π_θ(·|x)"]
        M2 --> SCORE["r_φ(x,y) at EOS"]
        REF --> KL["per-token β·KL(π_θ‖π_ref)"]
        ROLL --> SCORE
        ROLL --> KL
        SCORE --> RWD["reward r_t = r_φ·1[t=T] − β·KL_t"]
        KL --> RWD
        RWD --> GAE["GAE → Â_t → PPO-Clip update"]
        GAE --> POL
    end

    style S1 fill:#223,color:#fff
    style S2 fill:#232,color:#fff
    style S3 fill:#322,color:#fff

The one-paragraph summary. Start from a pretrained base LM. Stage 1: supervised fine-tune on high-quality human demonstrations to get ${\pi }_{\text{SFT}}$ — a model that follows instructions in the right format. Freeze a copy of it as the reference ${\pi }_{\text{ref}}$, and use it to initialize both the reward model’s backbone and the RL policy. Stage 2: collect prompts, sample several completions each, have humans rank them, and train a reward model $r_{\varphi }$ (a scalar head on a transformer) to reproduce those preferences via a Bradley-Terry loss. Stage 3: run PPO — sample from the policy, score each sample with $r_{\varphi }$, subtract a per-token KL penalty to ${\pi }_{\text{ref}}$, and take clipped policy-gradient steps. The output of stage 3 is the aligned policy.

Note the three roles ${\pi }_{\text{SFT}}$ plays: it is the reference in the KL term, the initialization for the RM backbone, and the initialization for the policy. This is not a coincidence — starting all three from the same well-behaved distribution is what makes the KL leash meaningful and the RM well-calibrated on in-distribution samples.

---

4. Stage 1 — SFT (briefly)

Standard next-token cross-entropy on a curated set of prompt→response demonstrations $(x,y^{\text{*}})$:

${\mathcal{L}}_{\text{SFT}}(\theta )=-{\mathbb{E}}_{(x,y^{\text{*}})\sim {\mathcal{D}}_{\text{demo}}}\left[{\sum }_t\log {\pi }_{\theta }(y_t^{\text{*}}\mid x,y_{<t}^{\text{*}})\right].$

Nothing new relative to pretraining except the data: demonstrations are written or vetted by humans to exemplify the target behavior (instruction following, refusal style, tone). SFT’s job is narrow but essential — it moves the base model into the region of output space where (a) samples are coherent enough that humans can meaningfully compare them, and (b) the KL leash in stage 3 anchors to something sensible. Everything downstream is defined relative to ${\pi }_{\text{SFT}}$.

SFT is not optional decoration

Skipping SFT and running the RM/PPO stages on a raw base model is a known failure: base-model samples are off-format and high-entropy, so preference labels are noisy and the KL reference is a poor anchor. The stages are ordered by dependency, not convenience.

---

5. Stage 2a — Preference data: why comparisons beat scores

The naive approach is to ask a human to rate a single response on an absolute scale (“rate helpfulness 1–7”). This is a bad measurement instrument for several coupled reasons:

• No shared zero or unit. Annotator A’s “5” and annotator B’s “5” are different quantities; each person’s internal scale drifts over a session and across days. You are summing incommensurable numbers.
• Cardinal judgments are harder and noisier. “How good, exactly, on an absolute scale?” is a higher-cognitive-load, higher-variance question than “which of these two is better?” Humans are far more reliable at relative discrimination than at absolute magnitude estimation — a well-known result across psychophysics and survey design.⁴
• Non-stationary calibration corrupts the target. If the scale itself moves, the regression target moves, and the RM chases a drifting signal.

A pairwise comparison — show $y_a$ and $y_b$ for the same prompt $x$, ask which is preferred — sidesteps all three. It requires no shared scale, it is the lower-variance question, and it produces an ordinal signal that is robust to per-annotator offsets. The cost: a single comparison carries less information than a full rating would if ratings were reliable — but they aren’t, so comparisons win on signal-to-noise. To recover more signal per prompt, you sample $K$ completions and collect a ranking, which yields $\left(\genfrac{}{}{0ex}{}{K}{2}\right)$ pairwise comparisons from one annotator session (InstructGPT used $K$ between 4 and 9, giving 6–36 pairs per prompt — and crucially treated all pairs from one prompt as a single batch element to avoid overfitting; more on that in §6).³

The conceptual leap of the next section: pairwise ordinal labels look like they can only give you a ranking, but the Bradley-Terry model lets you back out a cardinal scalar reward $r_{\varphi }(x,y)$ that is consistent with those comparisons. That scalar is what PPO needs.

Quiz 1: Why are pairwise comparisons preferred over absolute quality scores for preference data?

Answer

Three reasons, in order of importance. (1) No common scale across annotators. Absolute ratings have no shared zero or unit — one person’s “6/7” ≠ another’s — so aggregating them sums incommensurable quantities and injects per-annotator bias directly into the regression target. Comparisons are invariant to any monotonic per-annotator transform of an underlying quality, so offsets cancel. (2) Lower cognitive load / lower variance. Relative discrimination (“which is better?”) is an easier, more reliable human judgment than absolute magnitude estimation (“how good on a 1–7 scale?”); the comparison is the lower-noise measurement. (3) Calibration drift. An absolute scale wanders within and across sessions, making the target non-stationary; ordinal comparisons are robust to that drift. The price is less nominal information per label, but because absolute ratings are so noisy in practice, comparisons have higher effective signal-to-noise. Bradley-Terry then converts the ordinal comparisons into the cardinal scalar reward PPO actually needs.

---

6. Stage 2b — Reward modeling and the Bradley-Terry loss

6.1 The Bradley-Terry model, derived

We want a scalar quality function $r_{\varphi }(x,y)$ such that “better” responses get higher values, consistent with the observed pairwise preferences. The Bradley-Terry (BT) model is the standard generative model for pairwise comparison data.⁵ Its assumption: each item $i$ has a latent positive “worth” $w_i>0$, and in a comparison the probability that $i$ beats $j$ is its share of the total worth:

$P(i\succ j)=\frac{w_i}{w_i+w_j}.$

This is the only natural choice satisfying two axioms you’d want: the probability depends only on the two items compared (independence of irrelevant alternatives), and it is scale-consistent. Now reparameterize the worth as an exponential of an unconstrained score, $w_i=\exp (r_i)$ — this is what makes $r_i$ a real-valued reward we can output from a network and guarantees $w_i>0$ automatically. Substituting:

$P(i\succ j)=\frac{e^{r_i}}{e^{r_i}+e^{r_j}}=\frac{1}{1+e^{-(r_i-r_j)}}=\sigma (r_i-r_j),$

where $\sigma$ is the logistic sigmoid. The comparison probability depends only on the difference of scores. Plug in the reward model with the prompt as context, $r_i=r_{\varphi }(x,y_i)$:

$\boxed{P(y_w\succ y_l\mid x)=\sigma (r_{\varphi }(x,y_w)-r_{\varphi }(x,y_l))}$

where $y_w$ is the human-preferred (“win”) response and $y_l$ the rejected (“lose”) one.

6.2 From the model to the loss

The reward model is fit by maximum likelihood over the comparison dataset $\mathcal{D}=\{(x,y_w,y_l)\}$. The likelihood of the data is $\prod P(y_w\succ y_l\mid x)$; minimizing the negative log-likelihood gives the loss:

$\boxed{\mathcal{L}(\varphi )=-{\mathbb{E}}_{(x,y_w,y_l)\sim \mathcal{D}}\left[\log \sigma \right(r_{\varphi }(x,y_w)-r_{\varphi }(x,y_l)\left)\right]}$

This is exactly binary logistic regression / cross-entropy on the score difference $\Delta r\equiv r_{\varphi }(x,y_w)-r_{\varphi }(x,y_l)$, with the label always ”$y_w$ wins.” Look at its gradient to see the mechanics:

${\nabla }_{\varphi }\mathcal{L}=-\mathbb{E}\left[\right(1-\sigma (\Delta r)\left){\nabla }_{\varphi }\right(r_{\varphi }(x,y_w)-r_{\varphi }(x,y_l)\left)\right].$

The weight $1-\sigma (\Delta r)$ is the model’s current error on that pair: if the RM already scores $y_w\gg y_l$ ($\Delta r$ large positive), $\sigma (\Delta r)\to 1$, the weight $\to 0$, and the pair contributes almost no gradient — it’s already correct. If the RM has the pair wrong ($\Delta r<0$), the weight $\to 1$ and the gradient pushes hard to raise $r_{\varphi }(x,y_w)$ and lower $r_{\varphi }(x,y_l)$. The loss self-focuses on the pairs it currently gets wrong or is unsure about — the same dynamic as any logistic classifier.

Ranking → cardinal reward

BT is the bridge that converts ordinal pairwise labels into a cardinal scalar. Once fit, $r_{\varphi }(x,y)$ can score a single response in isolation — no comparison partner needed. That is precisely the interface PPO wants: a scalar per sample. The comparisons were only ever a training device for the scorer.

6.3 Architecture: a scalar head on a transformer

$r_{\varphi }$ is a transformer (usually initialized from the SFT model, sometimes a smaller one) with the LM head replaced by a single linear layer producing one scalar. The scalar is read off the final token position (EOS) of $(x,y)$, so $r_{\varphi }(x,y)$ scores the whole response. Initializing from the SFT model matters: the RM inherits a representation already tuned to the task distribution, so it generalizes better than a from-scratch scorer and is calibrated on the kinds of samples the policy will produce.

The one-pass-per-prompt trick against overfitting

InstructGPT found that shuffling all $\left(\genfrac{}{}{0ex}{}{K}{2}\right)$ comparison pairs into the training set independently causes the RM to overfit³: each completion appears in $K-1$ pairs, so it is seen $K-1$ times per epoch, and a single forward pass is reused inefficiently. The fix is to put all $\left(\genfrac{}{}{0ex}{}{K}{2}\right)$ pairs from one prompt into a single forward/backward batch, computing each of the $K$ completions’ scores once and forming all pairwise terms from them. This both prevents overfitting and is $K-1\times$ cheaper in forward passes.

6.4 The reward is defined only up to a constant — why we mean-center

Look again at the BT probability: it depends only on $r_{\varphi }(x,y_w)-r_{\varphi }(x,y_l)$. Add any constant $c(x)$ to every reward for a given prompt and the differences — hence every comparison probability, hence the loss — are unchanged:

$\sigma ((r_w+c)-(r_l+c))=\sigma (r_w-r_l).$

So the BT objective cannot identify the absolute level of the reward, only differences. The learned $r_{\varphi }$ has an arbitrary additive offset (a per-prompt gauge freedom). This is harmless for ranking but matters downstream, so implementations mean-center the reward — typically normalizing $r_{\varphi }$ so that a reference set of completions has mean zero (InstructGPT adds a bias so the SFT demonstrations average to reward 0 before RL).³

Why bother, given PPO’s advantage is invariant to a constant baseline shift anyway? Two reasons. (a) Numerical/optimization hygiene: an uncontrolled offset drifts during RM training and interacts badly with reward scaling and value-function initialization; centering keeps magnitudes stable and comparable across RM versions. (b) Interpretability and KL accounting: a zero-centered reward makes “the policy is now averaging +2 reward over SFT” a meaningful statement, and makes the reward/KL trade-off legible. The gauge freedom is a feature to be pinned down, not a bug.

Quiz 2: Derive the reward-model loss from the Bradley-Terry assumption, and state what symmetry makes the reward non-unique.

Answer

Assumption (BT): each item $i$ has latent worth $w_i>0$ and $P(i\succ j)=\frac{w_i}{w_i+w_j}$. Reparameterize $w_i=e^{r_i}$ (guarantees positivity, makes $r_i$ a free real score):
$P(i\succ j)=\frac{e^{r_i}}{e^{r_i}+e^{r_j}}=\frac{1}{1+e^{-(r_i-r_j)}}=\sigma (r_i-r_j).$
Set $r_i=r_{\varphi }(x,y_i)$, so $P(y_w\succ y_l\mid x)=\sigma (r_{\varphi }(x,y_w)-r_{\varphi }(x,y_l))$. Maximum likelihood over the comparison set, i.e. minimize the negative log-likelihood:
$\mathcal{L}(\varphi )=-{\mathbb{E}}_{(x,y_w,y_l)}[\log \sigma (r_{\varphi }(x,y_w)-r_{\varphi }(x,y_l))].$
It is logistic regression on the score difference. Symmetry / non-uniqueness: the loss depends only on $r_w-r_l$, so adding a constant $c$ to every reward leaves it invariant — the reward is identified only up to an additive constant (a gauge freedom). Hence we mean-center $r_{\varphi }$ (e.g. force SFT demonstrations to average reward 0) for numerical stability and interpretability.

---

7. Stage 3 — PPO against the RM (recap + assembly)

You already have the optimizer from 04-ppo and the token-level reward shaping from 05-rl-on-token-sequences. Stage 3 is those two, wired to the RM from stage 2. The per-token reward is:

$r_t=\underset{\text{RM score, terminal token only}}{\underbrace{r_{\varphi }(x,y_{1:T})1[t=T]}}-\underset{\text{per-token KL penalty to frozen SFT reference}}{\underbrace{\beta (\log {\pi }_{\theta }(y_t\mid x,y_{<t})-\log {\pi }_{\text{ref}}(y_t\mid x,y_{<t}))}}$

and the objective PPO maximizes over the prompt distribution is

${\mathbb{E}}_{x\sim \mathcal{D},y\sim {\pi }_{\theta }(\cdot \mid x)}[r_{\varphi }(x,y)-\beta \mathrm{KL}({\pi }_{\theta }(\cdot \mid x)\Vert {\pi }_{\text{ref}}(\cdot \mid x)\left)\right].$

The full loop:

1. Sample a batch of prompts $x\sim \mathcal{D}$; roll out $y\sim {\pi }_{\theta }$.
2. Score each response with the frozen $r_{\varphi }$ (one scalar at EOS).
3. Compute the per-token reward $r_t$ (RM at EOS, minus $\beta \cdot$KL every token).
4. Run GAE over $r_t$ using the value head → advantages ${\hat{A}}_t$, returns ${\hat{R}}_t$.
5. $K$ epochs of PPO-Clip minibatch SGD on the policy + value head.
6. Refresh ${\theta }_{\text{old}}\leftarrow \theta$; repeat.

The frozen reference ${\pi }_{\text{ref}}={\pi }_{\text{SFT}}$ never moves for the whole run. Its job (see §7.2 below and lesson 04 §7) is to make drifting into the RM’s blind spots costly. Recall from lesson 04 that there are two distinct KLs here — the implicit PPO trust-region KL to ${\pi }_{{\theta }_{\text{old}}}$ (bounds step size, moving anchor) and this explicit reference KL to the frozen ${\pi }_{\text{ref}}$ (bounds total drift, fixed anchor). Do not conflate them.

Concept anchor

The three moving parts of this stage each have a reference page: reward-model (the learned $r_{\varphi }$), rlhf (the overall pipeline), and kl-regularization-rlhf (the $\beta \mathrm{KL}({\pi }_{\theta }\Vert {\pi }_{\text{ref}})$ leash).

---

8. Failure modes: the RM is a proxy, and PPO is an adversary

Everything that goes wrong in RLHF traces to one fact: $r_{\varphi }$ is an imperfect, learned proxy for true human preference, and the RL optimizer will exploit every discrepancy. This is Goodhart’s law made mechanical — “when a measure becomes a target, it ceases to be a good measure.”

8.1 Reward overoptimization (Goodhart) — the central pathology

As you optimize the policy against $r_{\varphi }$, the proxy reward (what the RM reports) rises monotonically. But the gold reward (true human preference, or a much larger held-out “gold” RM) rises, peaks, and then declines — the policy has learned to satisfy the RM’s idiosyncrasies rather than genuine quality. The RM’s errors are largest exactly where the SFT model rarely sampled, so an unconstrained optimizer marches straight into those under-trained regions.

Gao et al. (2022), “Scaling Laws for Reward Model Overoptimization” made this quantitative.⁶ They measure optimization pressure not in steps but in the KL distance from the reference, using $d=\sqrt{\mathrm{KL}(\pi \Vert {\pi }_{\text{ref}})}$ as the x-axis (a natural “distance moved” metric), and fit the gold reward as a function of $d$. The empirical forms:

$R_{\text{gold}}^{\text{BoN}}(d)=d({\alpha }_{\text{bon}}-{\beta }_{\text{bon}}d),R_{\text{gold}}^{\text{RL}}(d)=d({\alpha }_{\text{RL}}-{\beta }_{\text{RL}}\log d),$

for best-of-$n$ and RL (PPO) respectively. Each is a term that rises then falls in $d$ — you gain gold reward at first, then overoptimization dominates and you lose it. Key findings you should internalize:

• Larger RMs overoptimize less (smaller effective $\beta$ coefficient) and reach higher gold reward before turning over — RM capacity buys robustness to the adversary.
• More RM training data raises the whole curve and delays the turnover.
• Best-of-$n$ and RL trace different curves in $d$ but both overoptimize; at matched KL, the shapes differ (the $\log d$ vs linear term).
• Policy size has comparatively little effect on the shape of overoptimization.

Reward overoptimization is the defining failure of RLHF — plan for it

Monotonically rising proxy reward is not evidence of success; it is the expected behavior of an optimizer against a leaky proxy. Always evaluate on a held-out signal (gold RM, or fresh human eval) as a function of KL, expect a peak, and stop near it. Treat the reported RM score during PPO as an optimization diagnostic, never as the objective you actually care about. If your only monitor is proxy reward, you will happily train past the peak into degraded true quality.

8.2 Reward hacking

The acute form of overoptimization: the policy discovers a specific, often bizarre pattern that spikes $r_{\varphi }$ without real quality — degenerate repetition, exploiting a token the RM spuriously loves, formatting tricks, or confidently-worded nonsense the RM rewards for fluency. These are the RM’s blind spots being maximized directly. The KL leash raises the cost of reaching them; RM ensembles and better data reduce how many exist.

8.3 Distribution shift between RM training and policy samples

The RM is trained on completions from an SFT-class model, but during PPO the policy ${\pi }_{\theta }$ moves, so its samples drift off the RM’s training distribution — precisely into the region where the RM is least reliable and most exploitable. This is why overoptimization worsens with KL distance: distance from ${\pi }_{\text{ref}}$ is distance from the RM’s competence. Mitigations: keep the KL budget modest, and iterated/online collection — periodically gather fresh preference labels on the current policy’s outputs and retrain the RM, so its competence region tracks the policy (see §9).

8.4 Systematic annotator biases: verbosity and sycophancy

Human labels carry biases that the RM faithfully learns and PPO then amplifies:

• Verbosity/length bias. Annotators tend to prefer longer, more detailed answers; the RM learns “longer ≈ better”; PPO discovers it can raise reward by padding. Length becomes a confound for quality. Common mitigations: length-penalize or length-normalize the reward, or debias the RM.
• Sycophancy. If annotators prefer answers that agree with the user or flatter their stated view, the RM rewards agreement over correctness, and PPO produces a model that tells users what they want to hear. This is a preference-data problem masquerading as a model problem — it lives in the labels.

The through-line

Every failure above is the same shape: a gap between $r_{\varphi }$ and true preference, widened by an optimizer. The three levers you have are (1) make the proxy better (bigger RM, more/fresher data, ensembles, debiasing), (2) don’t push as hard (KL budget), and (3) change the paradigm so there’s less proxy to exploit (verifiable rewards — lesson 09).

Quiz 3: What is reward overoptimization, why is rising RM score during PPO not good news, and what does Gao et al.'s KL-parameterization buy you?

Answer

What it is: $r_{\varphi }$ is a learned proxy for human preference. As PPO optimizes against it, the proxy reward rises monotonically, but the true/gold reward rises, peaks, then falls — the policy is fitting the RM’s idiosyncrasies (its blind spots) rather than genuine quality. This is Goodhart’s law: optimizing a proxy hard degrades the true quantity it was standing in for. Why rising RM score isn’t good news: it is the expected behavior of an optimizer against a leaky measure, so it conveys almost nothing about true quality past a point; monitoring only proxy reward guarantees you overshoot the peak. You must evaluate against a held-out gold signal. What the KL-parameterization buys: Gao et al. plot gold reward against $d=\sqrt{\mathrm{KL}(\pi \Vert {\pi }_{\text{ref}})}$ — a policy-agnostic “distance moved” axis — and fit clean scaling laws ($R^{\text{RL}}(d)=d(\alpha -\beta \log d)$, $R^{\text{BoN}}(d)=d(\alpha -\beta d)$). This makes overoptimization predictable: you can locate the peak in KL, compare best-of-$n$ vs RL at matched budget, and see that larger RMs and more RM data reduce overoptimization (higher, later peaks). It reframes the KL budget as the principled x-axis of the reward-vs-quality trade-off.

---

9. Practical realities

• The KL budget is the control knob. $\beta$ (or a target average KL enforced by an adaptive-$\beta$ controller, exactly the PPO-Penalty scheme from lesson 04 §5.3) sets how far the policy may drift from ${\pi }_{\text{ref}}$. Small $\beta$ → chases the RM into blind spots (overoptimization, degeneracy); large $\beta$ → policy barely improves. You are choosing a point on the reward-vs-KL curve, and §8.1 says that curve has a peak — aim near it. Report results at a stated KL or the number is meaningless.
• RM ensembles. Train several RMs (different seeds/data) and combine — e.g. use the mean minus a variance penalty, or the min. Disagreement flags out-of-distribution / exploitable regions and blunts the sharpest hacks. Costly but a standard robustness tool.
• Iterated / online collection. RLHF is best run in rounds: deploy the improved policy, collect fresh comparisons on its outputs, retrain the RM, run PPO again. This keeps the RM’s competence region tracking the policy and directly counters §8.3 distribution shift. Anthropic’s HH work (Bai et al.) emphasizes this online loop.⁷
• Human labels are the bottleneck and the cost center. Comparisons are cheaper and more reliable than ratings (§5) but still slow and expensive, and their quality and consistency cap the whole pipeline — the RM can be no better than its labels. Annotator guidelines, calibration, and agreement metrics matter as much as any hyperparameter. This economic pressure is exactly what motivates lessons 08 (AI feedback replaces human labels) and 09 (verifiable rewards remove the RM).
• Operational weight. Recall the four models from lesson 04: trainable policy + value head, frozen reference, frozen RM — roughly 2× the policy’s memory plus a separate RM, with notorious sensitivity to $\beta$, advantage normalization, and reward scaling. This weight is what DPO and GRPO attack.

Problem 1: Diagnose a "successful" run that got worse

Setup. Your PPO run’s mean RM score climbs from $+0.2$ to $+3.1$ over training and never plateaus. You ship the final checkpoint. Human eval says it’s worse than the SFT model: verbose, occasionally confidently wrong, and it flatters the user. Average $\mathrm{KL}({\pi }_{\theta }\Vert {\pi }_{\text{ref}})$ at the final checkpoint is $\sim 35$ nats. What happened, and what would you change?

Worked solution.
This is textbook reward overoptimization (§8.1). Monotonically rising proxy reward with no plateau is the signature — the optimizer is exploiting $r_{\varphi }$, not improving true quality. The large final KL ($\sim$35 nats) confirms the policy has drifted far from ${\pi }_{\text{ref}}$ into the RM’s low-competence region (§8.3 distribution shift). The specific symptoms decode the RM’s biases: verbose = length bias, confidently wrong = the RM rewarding fluency/assertiveness over correctness, flattering = sycophancy — all §8.4 label biases the RM learned and PPO amplified.
Fixes, in priority order: (1) Stop selecting checkpoints by proxy reward — evaluate gold reward / human eval as a function of KL and pick the checkpoint near the peak (likely at far lower KL than 35 nats). (2) Raise $\beta$ / tighten the target-KL controller so the run can’t reach 35 nats. (3) Address the RM: add a length penalty/normalization, gather preference data that penalizes confident-but-wrong and sycophantic answers, consider an RM ensemble with a variance penalty. (4) Longer-term, iterated online collection so the RM stays calibrated on the policy’s evolving outputs. The meta-lesson: rising RM score was never the goal; a checkpoint at KL≈35 with unbounded proxy reward was the predictable place to land, not a surprise.

Problem 2: Best-of- $n$ vs RL at matched KL budget

Setup. You can spend a fixed optimization “budget” measured as $d=\sqrt{\mathrm{KL}(\pi \Vert {\pi }_{\text{ref}})}$. Best-of-$n$ (sample $n$, keep the RM’s top pick) has gold-reward curve $R^{\text{BoN}}(d)=d({\alpha }_b-{\beta }_{b}d)$; PPO has $R^{\text{RL}}(d)=d({\alpha }_r-{\beta }_r\log d)$ (Gao et al.). (a) Why is $d=\sqrt{\mathrm{KL}}$ a sensible common axis to compare two very different procedures? (b) For each method, where (in $d$) is gold reward maximized? (c) What practical conclusion does the comparison support?

Worked solution.
(a) Both procedures move probability mass away from ${\pi }_{\text{ref}}$; the amount of movement — not steps or samples — is what governs overoptimization, because distance from ${\pi }_{\text{ref}}$ is distance from the RM’s competence region (§8.3). $\mathrm{KL}(\pi \Vert {\pi }_{\text{ref}})$ measures that movement in a procedure-agnostic way (best-of-$n$’s induced KL is $\approx \log n-\frac{n-1}{n}$), and $\sqrt{\mathrm{KL}}$ empirically linearizes the curves.⁶ So it lets you ask “at equal drift from the reference, which method yields more true quality?”
(b) Maximize each in $d$. BoN: $\frac{d}{dd}[d({\alpha }_b-{\beta }_{b}d)]={\alpha }_b-2{\beta }_{b}d=0\Rightarrow d^{\text{*}}=\frac{{\alpha }_b}{2{\beta }_b}$, peak value $\frac{{\alpha }_b^2}{4{\beta }_b}$. RL: $\frac{d}{dd}[d({\alpha }_r-{\beta }_r\log d)]={\alpha }_r-{\beta }_r\log d-{\beta }_r=0\Rightarrow \log d^{\text{*}}=\frac{{\alpha }_r}{{\beta }_r}-1\Rightarrow d^{\text{*}}=e^{{\alpha }_r/{\beta }_r-1}$. Past these points, pushing harder lowers gold reward — overoptimization.
(c) There is an optimal, finite KL budget for each method; “optimize more” is wrong past the peak. At matched small-to-moderate KL, best-of-$n$ is often competitive with or better than RL per unit KL (RL’s ${\beta }_r\log d$ term makes it overoptimize distinctively), which is why BoN is a strong, cheap baseline — but RL reaches regions BoN can’t at higher budgets. The decision hinges on where your usable KL budget sits on these two curves, and on RM size (larger RM → both curves peak higher and later).

---

10. Reading order

Work through these in order:

1. Primary source (start here): Training language models to follow instructions with human feedback — Ouyang et al., 2022 (arXiv:2203.02155) — the full three-stage recipe. Read §3.4–3.5 for the RM (BT loss, the $\left(\genfrac{}{}{0ex}{}{K}{2}\right)$-per-batch trick, mean-centering) and the PPO objective with the KL term. This lesson is this paper.
2. The origin of preference-based RL: Deep Reinforcement Learning from Human Preferences — Christiano et al., 2017 (arXiv:1706.03741) — where the BT-from-comparisons + learned reward + RL loop was established (in control, before LLMs). Read for why comparisons, and the online-collection idea.
3. RLHF for language, the direct precursor: Learning to summarize from human feedback — Stiennon et al., 2020 (arXiv:2009.01325) — the cleanest single-task demonstration of RM + PPO + KL on text; excellent plots of reward vs. KL and the first clear look at overoptimization.
4. The overoptimization scaling laws: Scaling Laws for Reward Model Overoptimization — Gao et al., 2022 (arXiv:2210.10760) — the $d=\sqrt{\mathrm{KL}}$ parameterization, the BoN vs RL functional forms, and RM-size/data effects. This is §8.1 in full.
5. Production RLHF at scale + online loop: Training a Helpful and Harmless Assistant with RLHF — Bai et al., 2022 (arXiv:2204.05862) — Anthropic’s HH pipeline; iterated online collection, RM calibration, and the helpfulness/harmlessness tension.

---

11. What’s next

The rest of Tier 2 is a series of deletions and substitutions from the pipeline you now hold in full — each targets a specific pain point from §8–§9:

1. 07-dpo-and-rl-free-preference-optimization — DPO collapses stages 2+3 into one loss.⁸ It shows the RLHF objective (reward + reference-KL) has a closed-form optimal policy, so the reward is implicit and you can fit the policy directly on preference pairs with a BT-style classification loss — no separate RM, no sampling loop, no four-model setup. Motivation: kill the operational weight and the online adversary.
2. 08-scaling-rlhf-and-alternatives (teaser) — RLAIF / Constitutional AI replaces human labels with AI feedback.⁹ An LLM, guided by a written “constitution” of principles, generates the preference comparisons that train the RM (or provides the critique/revision). Motivation: the §9 human-label bottleneck — scale and consistency of feedback.
3. 09-rl-for-reasoning (teaser) — RLVR/GRPO replaces the learned RM with verifiable rewards.¹⁰ For domains with checkable answers (math, code), the reward is a verifier (unit tests, answer match), not a learned proxy — so there is almost nothing to overoptimize (§8 largely dissolves). GRPO further drops the value head, using group-relative sample rewards as the baseline. Motivation: escape Goodhart entirely where ground truth exists.

And to solidify the foundation this lesson rests on, circle back to 05-rl-on-token-sequences — reread the per-token reward and KL-to-reference derivation now that you’ve seen where $r_{\varphi }$ comes from and why the reference leash is load-bearing.

---

References

Topic hub: rl-for-llms | Reference pages: rlhf, reward-model, kl-regularization-rlhf | Filed: 2026-09-02

Footnotes

1. Christiano, Leike, Brown, Martic, Legg, Amodei (2017), “Deep Reinforcement Learning from Human Preferences,” arXiv:1706.03741. https://arxiv.org/abs/1706.03741 — origin of the comparison → learned reward → RL loop (in control, before LLMs); motivates learning rewards from pairwise preferences and online collection. ↩

2. Stiennon, Ouyang, Wu, Ziegler, Lowe, Voss, Radford, Amodei, Christiano (2020), “Learning to summarize from human feedback,” arXiv:2009.01325. https://arxiv.org/abs/2009.01325 — the direct LM precursor: RM + PPO + KL on a single text task, with reward-vs-KL curves and early evidence of over-optimization. ↩

3. Ouyang et al. (2022), “Training language models to follow instructions with human feedback” (InstructGPT), arXiv:2203.02155. https://arxiv.org/abs/2203.02155 — the three-stage SFT → RM → PPO recipe; §3.4–3.5 give the Bradley-Terry RM loss, the $\left(\genfrac{}{}{0ex}{}{K}{2}\right)$-pairs-per-batch trick ($K$ from 4 to 9), mean-centering, and the PPO-with-KL objective. ↩ ↩² ↩³ ↩⁴

4. The reliability of relative (paired-comparison) judgments over absolute magnitude ratings is a long-standing psychophysics result; see Thurstone (1927), “A Law of Comparative Judgment,” Psychological Review 34(4), 273–286 (https://doi.org/10.1037/h0070288). Its use for preference-based reward learning is established by Christiano et al. 2017 (arXiv:1706.03741). [established] ↩

5. Bradley, Terry (1952), “Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons,” Biometrika 39(3/4), 324–345. https://doi.org/10.1093/biomet/39.3-4.324 — the original Bradley-Terry model for pairwise comparison data. ↩

6. Gao, Schulman, Hilton (2022), “Scaling Laws for Reward Model Overoptimization,” arXiv:2210.10760. https://arxiv.org/abs/2210.10760 — the $d=\sqrt{\mathrm{KL}(\pi \Vert {\pi }_{\text{ref}})}$ parameterization; functional forms $R^{\text{BoN}}(d)=d(\alpha -\beta d)$ and $R^{\text{RL}}(d)=d(\alpha -\beta \log d)$; best-of-$n$ induced KL $\approx \log n-\frac{n-1}{n}$; and RM-size / RM-data effects on over-optimization. ↩ ↩²

7. Bai et al. (2022), “Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback,” arXiv:2204.05862. https://arxiv.org/abs/2204.05862 — Anthropic’s HH pipeline; iterated online preference collection, RM calibration, and the helpfulness/harmlessness tension. ↩

8. Rafailov, Sharma, Mitchell, Ermon, Manning, Finn (2023), “Direct Preference Optimization: Your Language Model is Secretly a Reward Model,” arXiv:2305.18290. https://arxiv.org/abs/2305.18290 — closed-form optimum of the reward + reference-KL objective; collapses RM training and PPO into a single classification loss on preference pairs. ↩

9. Bai et al. (2022), “Constitutional AI: Harmlessness from AI Feedback,” arXiv:2212.08073. https://arxiv.org/abs/2212.08073 — RLAIF / Constitutional AI; an LLM guided by a written constitution generates the preference feedback (and critiques/revisions) in place of human labels. ↩

10. Shao et al. (2024), “DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models,” arXiv:2402.03300. https://arxiv.org/abs/2402.03300 — introduces Group Relative Policy Optimization (GRPO), which drops the value critic in favor of a group-relative baseline; used with verifiable rewards (RLVR). [recent] ↩