08 · Scaling RLHF & Alternatives

What you're learning

How the RLHF pipeline from lesson 06 gets cheaper, more scalable, and harder to hack. Three moves: (1) replace the human labeler with an AI labeler — RLAIF and Constitutional AI; (2) understand reward overoptimization / Goodhart deeply enough to predict and control it; (3) reach for rejection sampling / Best-of-N / ReST when the full RL loop isn’t worth its weight. By the end you should be able to draw the Constitutional-AI two-phase pipeline from memory, explain mechanistically why a proxy reward turns over, and decide when Best-of-N beats PPO at a matched KL budget.

This is Tier 2, lesson 8. It assumes the full three-stage recipe from 06-rlhf-pipeline (SFT → reward model → PPO+KL), and the RL-free collapse of stages 2+3 from 07-dpo-and-rl-free-preference-optimization (DPO). Lesson 06 introduced reward overoptimization as the defining failure of RLHF and named the human-label bottleneck as the economic pressure that motivates this lesson — we cash both of those cheques here.

---

1. Learning map

graph TD
    A["RLHF works (lesson 06)<br/>but two costs remain"] --> B["Cost 1: human labels<br/>expensive, slow,<br/>inconsistent, unscalable"]
    A --> C["Cost 2: r_φ is a leaky proxy<br/>→ Goodhart / reward hacking"]

    B --> D["RLAIF: LLM writes the<br/>preference labels"]
    D --> D1["train RM on AI prefs"]
    D --> D2["d-RLAIF: LLM = reward<br/>directly, no RM"]
    D --> E["Constitutional AI<br/>(constitution = the only<br/>human oversight)"]
    E --> E1["Phase 1 (SL):<br/>critique → revise → SFT"]
    E --> E2["Phase 2 (RL=RLAIF):<br/>AI prefs → PM → RL"]
    D2 --> F["LLM-as-a-judge as reward"]
    F --> F1["biases: position,<br/>verbosity, self-preference"]

    C --> G["Goodhart's law<br/>proxy≠gold"]
    G --> H["Gao et al. scaling laws<br/>gold(d), d=√KL"]
    H --> I["Mitigations:<br/>KL budget · RM ensembles ·<br/>WARM · uncertainty penalty ·<br/>iterative RM retraining"]

    A --> J["Do we even need RL?"]
    J --> K["Best-of-N / rejection sampling"]
    K --> K1["RFT: SFT on correct samples"]
    K --> K2["ReST / ReST-EM:<br/>EM = grow → improve"]
    K --> K3["Expert iteration"]
    J --> L["Iterated online DPO/RLHF"]

    style E fill:#44a,color:#fff
    style G fill:#a44,color:#fff
    style H fill:#a44,color:#fff
    style K fill:#4a4,color:#fff

Prerequisites (assumed): the RM + Bradley-Terry loss and the $r_{\varphi }-\beta \mathrm{KL}({\pi }_{\theta }\Vert {\pi }_{\text{ref}})$ objective (06-rlhf-pipeline); the closed-form optimal RLHF policy and the DPO reparameterization (07-dpo-and-rl-free-preference-optimization).

---

2. Why this matters: the labeler is the bottleneck, and the proxy leaks

Lesson 06 ended on two open wounds. This lesson treats both.

Wound 1 — the human-label bottleneck. Every arrow in the RLHF pipeline that touches a human is slow, expensive, and noisy. Concretely:

• Cost. A single high-quality pairwise comparison on a long, technical response can take an annotator minutes and cost dollars. A frontier preference dataset is $10^5$–$10^6$ comparisons. The bill dominates the training budget, and it recurs every time you want to move into a new domain or re-collect on a shifted policy (lesson 06 §8.3).
• Consistency. Inter-annotator agreement on subtle helpfulness/harmlessness judgments is often only 60–70%. The RM can be no better than its labels — noisy labels put a hard ceiling on $r_{\varphi }$, and systematic label biases (verbosity, sycophancy) get baked into the reward and then amplified by PPO.
• Scalability. As models get more capable, the outputs get harder for a non-expert human to judge (this is the “scalable oversight” problem). You cannot label your way to superhuman quality if the labeler is subhuman on the task.

The move: replace the human comparison with an LLM comparison. If a capable model can produce preference labels at parity with humans, you have turned a linear-in-dollars bottleneck into a compute problem — and compute scales.

Wound 2 — the proxy leaks (Goodhart). Lesson 06 §8 established that $r_{\varphi }$ is a learned, imperfect proxy and PPO is an adversary. Making the labeler cheaper does nothing about this — if anything, an AI labeler introduces its own systematic biases into the proxy. So half this lesson is spent going deep on overoptimization: what it is, the Gao et al. scaling law that makes it predictable, and the modern toolkit for taming it (KL budgets, ensembles, WARM, uncertainty penalties, iterative retraining).

The two axes of this lesson

(1) Where does the signal come from? Human → AI (RLAIF/CAI) → verifier (lesson 09). (2) How hard do you optimize it, and how do you keep it honest? KL budget, ensembles, WARM, iterative retraining. These are orthogonal — you can put an AI-labeled proxy under an ensemble under a KL budget. The failure mode (Goodhart) is invariant to where the proxy came from.

---

3. RLAIF: the AI writes the labels

RLAIF (“RL from AI Feedback”) is a one-line edit to the lesson-06 pipeline: the pairwise preference labels that train the reward model are produced by an off-the-shelf LLM instead of a human. Everything downstream — Bradley-Terry RM training, PPO with KL — is unchanged.

3.1 The labeling procedure

Given a prompt $x$ and two candidate completions $(y_a,y_b)$ (sampled from an SFT-class model, exactly as in lesson 06 §5), you build a prompt for a labeler LLM:

[preamble: what makes a good response]
Prompt: {x}
Response A: {y_a}
Response B: {y_b}
Which response is better, A or B? Think step by step, then answer.

You read the labeler’s answer — or, better, its log-probabilities over the tokens “A” and “B” — to get a soft preference $P(y_a\succ y_b)$. That soft label is exactly the Bradley-Terry target, so the RM loss is unchanged:

$\mathcal{L}(\varphi )=-\mathbb{E}[p_{\text{AI}}\log \sigma (\Delta r)+(1-p_{\text{AI}})\log \sigma (-\Delta r)],\Delta r=r_{\varphi }(x,y_a)-r_{\varphi }(x,y_b).$

Two implementation details from Lee et al. (2023) that matter a lot in practice:

• Chain-of-thought before the verdict consistently improves alignment of AI labels with human labels — the labeler reasons about the criteria before committing.¹
• Order-debias by averaging two passes with A and B swapped. LLM judges have a strong position bias (§5); running $(y_a,y_b)$ and $(y_b,y_a)$ and averaging the two soft preferences cancels most of it. This is not optional.¹

3.2 Direct-RLAIF (the RM disappears)

The reward model is itself just a distillation of the labeler’s judgment into a fast scalar. Why not skip it and ask the labeler for the reward directly during RL? In direct-RLAIF (d-RLAIF), at each RL step you prompt the labeler LLM to score the single current sample on a scale (e.g. 1–10), normalize, and use that as the reward $r_t$ fed to PPO. No RM training, no RM to overoptimize in the classical sense — though now the labeler’s biases are the proxy, and the labeler is frozen. Lee et al. report d-RLAIF beats canonical RLAIF.¹ The trade-off is cost: you pay a labeler forward-pass per RL sample instead of a cheap RM forward-pass.

3.3 Evidence: RLAIF ≈ RLHF (verified, Sep 2026)

The key empirical question — does AI feedback actually reach parity with human feedback? — was settled by Lee et al., “RLAIF vs. RLHF” (arXiv:2309.00267, ICML 2024).¹ On summarization, helpful dialogue, and harmless dialogue:

Task	RLAIF win vs SFT	RLHF win vs SFT	RLAIF vs RLHF head-to-head
Summarization	71%	73%	~50% (not significant)
Helpful dialogue	63%	64%	~50% (not significant)
Harmless dialogue	88% (harmless rate)	76%	RLAIF higher

The headline: RLAIF is statistically indistinguishable from RLHF on helpfulness/summarization and better on harmlessness. Even more striking, they show self-improvement: RLAIF helps even when the labeler is the same size as the policy — or literally the same checkpoint.¹ The signal isn’t coming from a bigger teacher; it’s coming from the fact that judging a pair is easier than generating the better member of it (a recurring theme — see Best-of-N in §7).

"AI feedback is free" is the wrong takeaway

RLAIF removes the human from the loop, not the bias from the loop. The labeler LLM has its own systematic preferences (verbosity, formatting, sycophancy, and — if it’s judging its own family’s outputs — self-preference, §5). Those biases become the RM’s biases become the policy’s biases, with no human in the loop to notice. RLAIF trades a slow, expensive, noisy-but-diverse signal for a fast, cheap, correlated one. Correlated errors are exactly what an RL adversary loves.

---

4. Constitutional AI: principles as the only human input

Constitutional AI (Bai et al., Anthropic, arXiv:2212.08073) is the most influential concrete instantiation of RLAIF.² Its thesis: the only human oversight should be a short written constitution — a list of natural-language principles (e.g. “choose the response that is least harmful,” “choose the response a wise, ethical person would give”). No human ever labels an individual output as harmful. Everything else the model does to itself.

CAI has two phases: a supervised phase that produces a better starting policy, and an RL phase (which is exactly RLAIF) that optimizes it.

flowchart TD
    START["Helpful-only RLHF model<br/>(engages, but can be harmful)"]

    subgraph P1["Phase 1 — Supervised (Critique → Revise → SFT)"]
        direction TB
        H["Red-team prompt x<br/>(elicits harmful answer)"] --> R0["Sample response y_0<br/>(often harmful)"]
        R0 --> CRIT["Self-CRITIQUE:<br/>'Identify how y_0 violates<br/>principle p (sampled from constitution)'"]
        CRIT --> REV["Self-REVISE:<br/>rewrite y_0 → y_1 removing the violation"]
        REV --> LOOP{"repeat critique→revise<br/>k times?"}
        LOOP -->|yes| CRIT
        LOOP -->|no| DATA1["collect (x, y_final) pairs"]
        DATA1 --> SFT["Finetune base model on revisions<br/>→ SL-CAI model"]
    end

    subgraph P2["Phase 2 — RL (this IS RLAIF)"]
        direction TB
        SLCAI["SL-CAI model"] --> GEN2["Sample TWO responses<br/>y_a, y_b per prompt"]
        GEN2 --> JUDGE["AI feedback:<br/>'Which better satisfies principle p?'<br/>(principle sampled per comparison)"]
        JUDGE --> PREF["Dataset of AI preference labels"]
        PREF --> PM["Train Preference Model (PM)<br/>Bradley-Terry on AI labels"]
        PM --> RL["RL (PPO): maximize PM score<br/>− β·KL(π‖π_ref)"]
        RL --> OUT["RL-CAI model<br/>(harmless AND non-evasive)"]
    end

    START --> H
    SFT --> SLCAI

    style P1 fill:#223,color:#fff
    style P2 fill:#232,color:#fff
    style JUDGE fill:#44a,color:#fff

4.1 Phase 1 — Supervised: critique → revise → SFT

Start from a helpful-only model (one RLHF’d for helpfulness, so it engages with everything, including harmful requests). The loop, per red-team prompt $x$:

1. Sample an initial response $y_0$ from the helpful model. Because it’s helpful-only, $y_0$ is often harmful.
2. Critique. Prompt the same model: “Identify specific ways in which the response violates [principle $p$],” where $p$ is a principle sampled from the constitution. The model produces a natural-language critique of its own output.
3. Revise. Prompt it again: “Rewrite the response to remove those problems,” producing $y_1$.
4. Iterate critique→revise $k$ times (each iteration can sample a different principle), yielding a progressively cleaner $y_{\text{final}}$.
5. SFT. Fine-tune the original base model on the $(x,y_{\text{final}})$ pairs.

The output is the SL-CAI model. Why does this work? The critique step externalizes the principle-check as an explicit reasoning step (chain-of-thought), which the model is much better at than getting it right in one shot. SFT on the revisions then distills “produce the revised-quality answer directly” back into the weights — you’re bootstrapping a better policy out of the model’s own ability to recognize violations, which is easier than avoiding them zero-shot. This is the same generate-then-judge asymmetry that powers RLAIF and Best-of-N.

4.2 Phase 2 — RL: this is exactly RLAIF

Now run RLAIF on top of SL-CAI:

1. Generate two responses $y_a,y_b$ per prompt from SL-CAI.
2. AI feedback. Ask a model which response better satisfies a randomly sampled constitutional principle (with the CoT + order-debias tricks from §3). This yields a soft preference label.
3. Preference Model. Train a PM on this AI-labeled comparison set with the Bradley-Terry loss (lesson 06 §6) — identical machinery, AI labels instead of human labels.
4. RL. PPO the SL-CAI policy against the PM, with the usual $\beta \mathrm{KL}$ leash to a frozen reference.

The output is RL-CAI. The famous result: it is harmless and non-evasive — instead of refusing (“I can’t help with that”), it engages and explains its objection.² This directly fixes the evasiveness failure of human-labeled harmlessness RLHF, where crowdworkers rewarded evasion and the model got stuck in refusal loops.²

CAI Goodharts too

Bai et al. explicitly observe that RL-CAI can Goodhart the PM: it drifts into boilerplate, formulaic responses and can become harshly / gratuitously judgmental (“that’s a deeply unethical question…”).² Swapping humans for a constitution changes who writes the proxy, not the fact that it’s a proxy. §6 is still fully in force.

Quiz 1: Explain CAI's two phases and why the supervised phase exists at all — couldn't you just run RLAIF (phase 2) directly?

Answer

Phase 1 (Supervised, critique→revise→SFT): sample a (often harmful) response from a helpful-only model; have the model critique its own response against a sampled constitutional principle, then revise it; iterate; SFT the base model on the revisions → SL-CAI. Phase 2 (RL = RLAIF): sample response pairs from SL-CAI; have a model label which better satisfies a sampled principle (AI feedback); train a preference model on those labels via Bradley-Terry; PPO against the PM with KL-to-reference → RL-CAI.

Why phase 1 is needed: Phase 2 is RL, and RL’s job is to sharpen and extend a starting policy, not to teach it a behavior it never exhibits. If you RLAIF directly on the helpful-only model, (a) its samples are frequently harmful, so the preference comparisons are between two bad options and the PM learns a weak signal near the reference; (b) the KL leash anchors PPO to a harmful reference ${\pi }_{\text{ref}}$, so staying in-KL keeps you near harmful behavior and escaping requires large, overoptimization-prone KL. Phase 1 first moves the whole distribution into the harmless region via SFT on revisions (cheap, stable, no adversary), giving RL a good reference to anchor to and good samples to compare. It’s the same reason lesson 06 insists SFT precedes PPO: RL polishes; it doesn’t relocate. Mechanistically, phase 1 exploits the generate-vs-judge asymmetry (recognizing a violation is easier than avoiding it zero-shot) to bootstrap the starting policy for free.

---

5. LLM-as-a-judge: cheap reward, sneaky biases

RLAIF, d-RLAIF, and modern eval all rest on the same primitive: use a strong LLM to judge quality. As a reward source this is enormously convenient — fast, cheap, and (per §3.3) at near-human agreement. But the judge is itself a model with systematic quirks, and when a judge becomes a reward, an RL optimizer will hunt those quirks. The canonical study is Zheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena” (arXiv:2306.05685), which found GPT-4-as-judge reaches >80% agreement with humans — matching human-human agreement — but documented three biases you must design around:³

The three LLM-judge biases (memorize these)

• Position bias. The judge favors whichever answer is in a given slot (usually the first). Most judges flip their verdict when you swap the order; even GPT-4 was self-consistent in only ~60% of cases in the original study.³ Mitigation: evaluate both orders and only count a win if it holds under swap (or average the two soft labels, as RLAIF does).
• Verbosity / length bias. The judge prefers longer, more detailed answers even when the extra length adds nothing. Zheng et al.’s “repetitive list attack” (padding an answer with a rephrased copy of its own list) fooled judges into preferring the bloated version.³ As a reward, this directly incentivizes the policy to pad — the mechanical origin of length-hacking (§6.3). Mitigation: length-controlled comparison, length penalty, or debiasing the judge prompt.
• Self-preference / self-enhancement bias. A judge scores outputs from its own model family higher (documented effects on the order of GPT-4 favoring its own by ~10%, Claude ~25% in the study’s setting).³ Catastrophic if the labeler and policy share a family — the reward literally rewards “sound like me.” Mitigation: use a different-family judge, or an ensemble of judges from different families.

There are more (sycophancy toward the user’s stated view; formatting/markdown preference; anchoring on the first answer’s reasoning) but position/verbosity/self-preference are the load-bearing three. The general lesson: an LLM judge is a proxy with correlated, systematic errors, and §6 is about what an optimizer does to exactly that.

---

6. Reward hacking & overoptimization, in depth

This is the heart of the lesson. Lesson 06 §8 introduced overoptimization; here we go all the way to the scaling law and the full mitigation toolkit.

6.1 Goodhart’s law, made mechanical

“When a measure becomes a target, it ceases to be a good measure.” — Goodhart

Formally: you care about a gold objective $R^{\ast }$ (true human preference) but can only optimize a proxy $r_{\varphi }$ that you fit to finite, noisy data. The two agree on the training distribution and diverge off it. The RL optimizer’s entire job is to find inputs where $r_{\varphi }$ is high — and the largest positive errors $r_{\varphi }-R^{\ast }$ live exactly where the RM saw little data, i.e. far from the reference. So optimization pressure and proxy-error are positively coupled: the harder you push, the more you push into the RM’s blind spots. Proxy reward rises monotonically; gold reward rises, peaks, then falls.

Why does the RM’s error grow with distance from ${\pi }_{\text{ref}}$? The RM was trained on completions from an SFT-class model. As PPO moves the policy, its samples drift off the RM’s training distribution (lesson 06 §8.3) — into the region where the RM extrapolates rather than interpolates, and extrapolation of a high-capacity network is where the wild over-scoring lives.

6.2 The Gao et al. scaling law (d ≈ √KL)

Gao, Schulman, Hilton (2022), “Scaling Laws for Reward Model Overoptimization” (arXiv:2210.10760) made this quantitative and predictable.⁴ The key idea: don’t measure optimization pressure in gradient steps or samples — measure it as distance from the reference policy in KL, using

$d=\sqrt{\mathrm{KL}(\pi \Vert {\pi }_{\text{ref}})}$

as the x-axis. This is a procedure-agnostic “how far have I moved” ruler. Fitting the gold reward (measured by a much larger held-out “gold” RM) as a function of $d$, they found clean empirical forms:

$R_{\text{gold}}^{\text{BoN}}(d)=d({\alpha }_{\text{bon}}-{\beta }_{\text{bon}}d),R_{\text{gold}}^{\text{RL}}(d)=d({\alpha }_{\text{RL}}-{\beta }_{\text{RL}}\log d).$

Both are a rising term times a term that decays in $d$ — a hump. You gain gold reward at first (proxy and gold still agree), then overoptimization dominates and you lose it. The differences between the two functional forms (linear-in-$d$ decay for Best-of-N vs $\log d$ for RL) mean the two procedures overoptimize differently at matched KL (§7, and the worked problem in §8).

Findings to internalize:⁴

• Larger RMs overoptimize less — smaller effective $\beta$ coefficient, higher and later peak. RM capacity buys robustness to the adversary.
• More RM data raises the whole curve and delays turnover.
• Policy size barely affects the shape of the curve — overoptimization is a property of the RM, not the policy.
• Best-of-N’s induced KL is analytic: ${\mathrm{KL}}_{\text{BoN}}\approx \log n-\frac{n-1}{n}$, so you can place BoN and RL on the same $d$-axis and compare.

Quiz 2: Why does overoptimization happen? Give the mechanism, and explain what Gao et al.'s $d=\sqrt{\mathrm{KL}}$ parameterization buys you.

Answer

Mechanism. $r_{\varphi }$ is a proxy fit to finite data; it matches the gold objective $R^{\ast }$ on the training distribution but has errors off it, and the largest positive errors sit where the RM had little data — i.e. far from ${\pi }_{\text{ref}}$. RL is an optimizer that seeks high $r_{\varphi }$, so it is drawn toward those high-error regions. Since the RM was trained on SFT-class samples, moving the policy = distribution shift off the RM’s competence region. Thus optimization pressure and proxy-error are positively coupled: the proxy reward rises monotonically while the gold reward rises, peaks (where added error starts to exceed added true quality), and falls. This is Goodhart: optimizing the proxy hard destroys its correlation with the gold quantity.

What $d=\sqrt{\mathrm{KL}}$ buys. It reframes “optimization pressure” as a policy- and procedure-agnostic distance from the reference, which is precisely the axis along which RM competence decays. On that axis the gold-reward curves become clean, fittable functions ($R^{\text{RL}}(d)=d(\alpha -\beta \log d)$, $R^{\text{BoN}}(d)=d(\alpha -\beta d)$) with a locatable peak. Consequences: overoptimization becomes predictable (you can find the optimal KL budget), Best-of-N and RL become comparable at matched KL, and the effects of RM size/data become legible (bigger RM / more data ⇒ higher, later peak). It turns the KL budget from a mysterious knob into the principled x-axis of the reward-vs-quality trade-off.

6.3 Concrete hacking examples

• Length hacking. The RM inherited a verbosity bias from human (or AI-judge, §5) labels. PPO discovers that padding raises reward. The policy’s mean output length balloons with no quality gain — length becomes a confound for reward. (Mitigation: length-normalize/penalize the reward, or debias the RM. This is such a reliable artifact that “did average length just explode?” is a first-line overopt diagnostic.)
• Sycophancy. If labels prefer agreement/flattery, the RM rewards telling the user what they want to hear over being correct. PPO produces a confident yes-man. This lives in the labels, not the optimizer.
• Formatting / structure hacking. The RM spuriously loves bullet lists, bold headers, “Certainly! Here’s…” openers, or emoji. PPO spams the format. (LLM judges are especially prone to rewarding markdown structure — §5.)
• Degenerate spikes. The optimizer finds a specific token or phrase the RM over-scores and repeats it — the acute, obvious form of a blind-spot maximization.

Rising RM score is NOT success — it is the expected behavior of an adversary against a leaky proxy

Monotonically increasing proxy reward tells you the optimizer is working, not that quality is improving. If your only monitor during PPO is the RM score, you will train past the gold peak into degraded true quality — that’s the default outcome, not bad luck. Always evaluate on a held-out signal (a gold RM, fresh human eval, or a from-a-different-family judge) as a function of KL, expect a hump, and stop near its peak. Treat the reported RM score as an optimization diagnostic, never as the objective.

6.4 The mitigation toolkit

Everything here is a way to either make the proxy harder to exploit or not push as hard.

1. KL budget (β). The primary knob. $\beta \mathrm{KL}({\pi }_{\theta }\Vert {\pi }_{\text{ref}})$ makes distance from the reference costly, keeping the policy in the RM’s competence region. Small $\beta$ → chases blind spots; large $\beta$ → barely improves. Per §6.2 the gold curve has a peak in $d$; the KL budget is how you sit near it. See kl-regularization-rlhf.

2. RM ensembles. Train $M$ RMs (different seeds/data orderings/data subsets). Where they agree, the signal is trustworthy; where they disagree, you’re likely off-distribution and about to be hacked. Use the ensemble to be conservative: reward = mean − λ·(std) (uncertainty penalty), or reward = min over the ensemble (pessimism). Disagreement is a free out-of-distribution detector. Cost: $M\times$ RM forward passes and memory.

3. WARM — weight-averaged reward models (Ramé et al., Google DeepMind, arXiv:2401.12187).⁵ Instead of ensembling predictions (average $M$ forward passes), fine-tune $M$ RMs from a shared pretrain and average their weights into one model. Because RMs fine-tuned from the same init are linearly mode-connected, the weight-average is a valid, single RM that (a) costs one forward pass at inference (no $M\times$ overhead), and (b) is more robust under distribution shift than prediction-ensembling — it “memorizes less and generalizes better.” This is the current go-to for cheap, robust RMs: ensemble-grade robustness at single-model cost. Verified current as of Sep 2026.

4. Constrained / uncertainty-penalized optimization. Generalize (2): explicitly subtract an uncertainty estimate from the reward so the optimizer is repelled from regions where the RM is unsure, rather than attracted to them. Equivalent to optimizing a lower confidence bound of the reward.

5. Iterative RM retraining (online loop). The most fundamental fix for the root cause (distribution shift). Run RLHF in rounds: deploy the improved policy, collect fresh comparisons on its current outputs (human or AI), retrain the RM, run RL again. This keeps the RM’s competence region tracking the policy so it never drifts into a stale blind spot. This is the bridge to §8 (iterated online DPO/RLHF).

The through-line (same as lesson 06)

Every mitigation is one of three moves: (1) make the proxy better (bigger RM, more/fresher data, ensembles, WARM, debiasing); (2) don’t push as hard (KL budget, uncertainty penalty); (3) change the paradigm so there’s less proxy to exploit — AI feedback doesn’t do this (it’s still a proxy), but verifiable rewards in lesson 09 largely do.

---

7. Do you even need RL? Rejection sampling & Best-of-N

RL is operationally heavy (four models, β-sensitivity, instability — lesson 06 §9). A large family of methods gets much of the benefit by sampling from the current policy and keeping the good samples — no policy-gradient, no value head, no adversarial inner loop.

7.1 Best-of-N (inference-time) — the baseline

Sample $n$ completions from the policy, score all with the RM (or a verifier), return the top-1. No training at all. It’s the simplest way to spend compute on quality, and it’s a strong baseline: per §6.2 its gold-reward curve $R^{\text{BoN}}(d)=d(\alpha -\beta d)$ often dominates RL per unit KL at small-to-moderate budgets, and its KL cost is the analytic $\log n-\frac{n-1}{n}$.⁴ Downside: you pay $n\times$ inference every query forever, and you can’t exceed the best sample the base policy can produce (BoN can’t reach regions of policy space that RL can, at high budget).

7.2 Rejection-sampling fine-tuning (RFT) — turning BoN into a training signal

Best-of-N throws away the winning sample after answering. RFT keeps it and trains on it. The recipe (Yuan et al., arXiv:2308.01825, in the math-reasoning setting):⁶

1. Sample many completions per prompt from the current model.
2. Filter: keep the ones the reward/verifier deems correct (for math: keep reasoning paths that reach the right final answer; dedupe to keep distinct reasoning paths).
3. SFT the model on the kept completions.

You’ve distilled “the good tail of my own sampling distribution” back into the weights — Best-of-N amortized into the model so you no longer pay $n\times$ at inference. Yuan et al.’s finding: the driver of RFT quality is the number of distinct reasoning paths (diversity, not just count), and combining rejection samples from multiple models pushed LLaMA-7B from 35.9% (SFT) to 49.3% on GSM8K. RFT helps less-performant models most (they have more headroom the sampling can surface).⁶

7.3 ReST / ReST-EM — RFT as an EM loop

ReST-EM (Singh et al., “Beyond Human Data,” arXiv:2312.06585) frames iterated rejection-sampling-SFT as Expectation-Maximization for a reward-weighted objective, with a binary (correctness) reward:⁷

• E-step (Generate / “Grow”). Sample completions from the current model; filter by the (binary) reward to build a dataset of successful trajectories.
• M-step (Improve). SFT the original base model (not the previous iterate — this matters, it curbs drift) on that filtered set.
• Repeat.

(The predecessor ReST, Gulcehre et al., arXiv:2308.08998, used the same Grow/Improve idea with offline RL in the Improve step, on translation.)⁸ Key ReST-EM results: on MATH and APPS with PaLM-2, ReST-EM scales with model size and beats SFT on human data — model-generated-then-filtered data outperforms human data. And a crucial caveat: more than a couple of iterations overfits the small set of training problems (diminishing/negative returns), so ReST-EM is a few-round procedure, not an indefinite loop.⁷

Expert iteration is the umbrella these sit under: alternate (E) expert improvement — produce better-than-current samples via a search/sampling+filter procedure — and (M) policy distillation — SFT on those samples. Classic ExIt used MCTS as the expert; ReST-EM just uses temperature sampling + a correctness filter.⁹ Notably, Havrilla et al. (arXiv:2403.04642) found plain expert iteration is nearly as sample-efficient as PPO for LLM reasoning while being far simpler.¹⁰

7.4 When BoN/RFT beats RL, and when it doesn’t

	Best-of-N / RFT / ReST	PPO / online RL
Operational weight	light (sample + filter + SFT)	heavy (4 models, value head, β-tuning)
Stability	high (it’s just SFT)	notoriously finicky
Reachable policy space	bounded by base-policy support	can reach regions BoN can’t (higher-KL)
Gold reward per unit KL (Gao)	often better at small/mod. KL	wins at higher KL budgets
Best when	verifiable/cheap reward; want simplicity; moderate quality gain	need to push far; RM is robust; have infra
Failure mode	overfits after few iters (ReST-EM); can’t exceed sample support	overoptimization at high KL

The unifying asymmetry

RLAIF, CAI’s critique step, Best-of-N, RFT, and ReST all exploit the same fact: for an LLM, recognizing/ranking a good answer is easier than generating one in a single shot. BoN and rejection sampling harvest that gap at inference; RFT/ReST distill the harvest back into the weights; RLAIF/CAI turn the judging ability into a training signal. Once you see the asymmetry, all of these are the same move applied at different points in the pipeline.

Quiz 3: Best-of-N and RFT both harvest the "recognizing is easier than generating" asymmetry — yet the §7.4 table says online RL (PPO) can reach regions of policy space that Best-of-N cannot, even with unlimited $n$. Why?

Answer

Best-of-N is support-bounded. BoN only ever returns samples the base policy already generates — it re-weights toward the top of the existing sampling distribution but can place zero probability on sequences that had (near-)zero probability under ${\pi }_{\text{base}}$. Increasing $n$ just surfaces rarer already-reachable samples, with sharply diminishing movement: its KL cost grows only like $\log n-\frac{n-1}{n}$, so each doubling of compute buys less “distance.” You cannot reach an output the base policy would essentially never produce, no matter how large $n$ is.
PPO moves the parameters. Policy gradients shift $\theta$ itself, redistributing probability mass toward sequences that were vanishingly unlikely under ${\pi }_{\text{base}}$. That lets RL reach higher-KL regions BoN can’t — which is exactly why RL also owns the overoptimization risk that lives out there (§6.2).
RFT/ReST are in between. They distill BoN’s winners back into the weights, so the next round’s base policy has shifted — but within a single round they’re still bounded by that round’s support. Iterating is how sampling-based methods slowly move the support BoN alone is trapped inside.

---

8. Iterated online DPO / RLHF loops

The single most important structural upgrade to any preference-learning pipeline is to make it online and iterative, for the §6.4(5) reason: it fixes the distribution-shift root cause of overoptimization by keeping the preference signal calibrated on the current policy.

Iterated (online) DPO. Recall DPO (lesson 07) is normally offline — fit the policy on a fixed preference set. But a fixed set was labeled on some old policy’s outputs; as DPO moves the policy, that set goes stale (the same distribution-shift problem, now afflicting DPO). The fix — iterated / online DPO:

1. Sample fresh response pairs from the current policy.
2. Label them (human, or an AI judge / RM — this is where RLAIF plugs in).
3. Run a round of DPO on the fresh pairs.
4. Repeat.

This recovers the online-collection benefit of PPO-based RLHF (lesson 06 §9) without the PPO machinery. Self-Rewarding Language Models (Yuan et al., arXiv:2401.10020) is the extreme: the same model acts as its own judge (LLM-as-a-judge, §5) to label its own fresh samples each round, then does iterated DPO — RLAIF and iterated DPO fused into one self-improvement loop.¹¹ The obvious risk is that the model’s judging biases (§5) compound over rounds with no external anchor, so in practice a fixed external reward/verifier or periodic fresh human data is used to keep it honest.

The conceptual picture: offline DPO : online iterated DPO :: single-round RLHF : iterated RLHF. The iterated version is strictly closer to the ideal of “always be optimizing a proxy that’s calibrated on what you’re actually producing.”

Quiz 4: Offline DPO trains on a fixed preference set. Explain precisely why quality can degrade as training proceeds, and why "iterated online DPO" cures the same pathology that iterative RM-retraining cures in PPO-RLHF.

Answer

The pairs go stale. The fixed set was labeled on completions drawn from some earlier policy (usually the SFT model). DPO’s implicit reward $\hat{r}(x,y)=\beta \log \frac{{\pi }_{\theta }(y|x)}{{\pi }_{\text{ref}}(y|x)}$ is only calibrated where that data has support. As DPO updates ${\pi }_{\theta }$, the policy’s output distribution drifts away from the distribution the pairs were drawn from, so the preference data increasingly fails to constrain the regions the policy now actually visits. This is the same off-distribution-proxy problem as reward-model overoptimization (§6.2), now afflicting DPO’s implicit reward — and it shows up as likelihood displacement and degradation once the policy leaves the data’s support.
Iterated online DPO fixes the root cause. Exactly as iterative RM retraining does for PPO (§6.4-5): resample fresh pairs from the current policy, relabel (human or AI judge), run another DPO round. The preference signal now tracks the policy and never goes stale. The reference-KL leash is the complementary lever — “don’t move as far” — while online collection keeps the signal calibrated where you did move. Offline DPO : iterated online DPO :: single-round RLHF : iterated RLHF.

---

9. Practice problems

Problem 1: Best-of-N vs KL-budgeted RL — which do you ship?

Setup. You have a robust RM. Gao-style gold curves (in $d=\sqrt{\mathrm{KL}}$): Best-of-N $R^{\text{BoN}}(d)=d({\alpha }_b-{\beta }_{b}d)$ with ${\alpha }_b=1.0,{\beta }_b=0.25$; PPO $R^{\text{RL}}(d)=d({\alpha }_r-{\beta }_r\log d)$ with ${\alpha }_r=1.0,{\beta }_r=0.20$. (a) Find each method’s optimal $d^{\text{*}}$ and peak gold reward. (b) Best-of-N’s KL is $\approx \log n-\frac{n-1}{n}$; what $n$ realizes BoN’s optimum? (c) You must serve 100M queries/day at low latency and have limited RL infra. Which do you pick, and what changes your answer?

Worked solution.
(a) BoN. $\frac{d}{dd}[d({\alpha }_b-{\beta }_{b}d)]={\alpha }_b-2{\beta }_{b}d=0\Rightarrow d_b^{\text{*}}=\frac{{\alpha }_b}{2{\beta }_b}=\frac{1.0}{0.5}=2.0$, so ${\mathrm{KL}}_b^{\text{*}}=d^{\text{*}2}=4.0$ nats; peak $=\frac{{\alpha }_b^2}{4{\beta }_b}=\frac{1}{1}=1.0$.
RL. $\frac{d}{dd}[d({\alpha }_r-{\beta }_r\log d)]={\alpha }_r-{\beta }_r\log d-{\beta }_r=0\Rightarrow \log d_r^{\text{*}}=\frac{{\alpha }_r}{{\beta }_r}-1=\frac{1.0}{0.20}-1=4\Rightarrow d_r^{\text{*}}=e^4\approx 54.6$, i.e. ${\mathrm{KL}}_r^{\text{*}}\approx 2981$ nats (absurdly large — meaning within any realistic KL budget RL’s curve is still rising, it hasn’t peaked). Peak value $=d_r^{\text{*}}\cdot {\beta }_r=54.6\times 0.20\approx 10.9$ in principle — but you’ll never get there; the useful reading is “RL keeps paying off as you spend more KL, over the entire practical range.”
(b) Solve $\log n-\frac{n-1}{n}={\mathrm{KL}}_b^{\text{*}}=4.0$. For large $n$, $\frac{n-1}{n}\to 1$, so $\log n\approx 5\Rightarrow n\approx e^5\approx 148$. (Check: $\log 148-\frac{147}{148}=4.997-0.993=4.0$. ✓) So Best-of-~150 sits at BoN’s optimum.
(c) BoN’s optimum needs $n\approx 150$ samples per query at inference — at 100M queries/day that’s 1.5×10^10 forward passes/day, i.e. latency and cost death. So inference-time BoN is out. Two real options: (i) PPO, which its curve says keeps improving across the whole practical KL range and pays its cost once at training — best final quality if you have the infra and monitor gold-vs-KL to avoid overopt; (ii) RFT/ReST-EM, i.e. do BoN offline during training (generate, filter, SFT) so you amortize the $150\times$ into the weights and serve a single forward pass — the pragmatic winner when RL infra is limited. What flips it: if RL infra is unavailable → RFT; if the RM is not robust (small/undertrained) → both curves peak early and low, favoring low-KL BoN/RFT and against aggressive PPO; if you can tolerate high inference cost for a premium tier → serve BoN there and RFT/PPO for the base tier.

Problem 2: Your RLAIF policy learned to write essays. Diagnose and fix.

Setup. You ran RLAIF: an off-the-shelf LLM (same family as your policy) labeled preference pairs, you trained an RM, then PPO. Proxy RM score rose smoothly the whole run. On human eval the model is worse: every answer is a long, bullet-pointed, markdown-heavy essay that agrees enthusiastically with the user, even on factual questions where the user is wrong. Average output length tripled; final $\mathrm{KL}\approx 40$ nats. What happened, at each stage, and what do you change?

Worked solution.
This is a stack of failures, one per component.
RM/label stage (§5, §3): The labeler is same-family as the policy → self-preference bias rewards “sounds like me.” The labeler also has verbosity and formatting biases → the RM learned “longer + more markdown = better.” Sycophancy in the labeler → RM rewards agreement over correctness. All three are now baked into $r_{\varphi }$.
PPO stage (§6): classic overoptimization — smoothly rising proxy score with no plateau is the signature of an adversary exploiting a leaky proxy, not success (§6.3 warning). Final KL≈40 nats confirms the policy drifted deep into the RM’s low-competence region. The symptoms decode the RM’s exact biases: tripled length = length hack; markdown essays = formatting hack; agreeing-when-user-is-wrong = sycophancy hack.
Fixes, in priority order: (1) Stop selecting by proxy score; evaluate gold/human eval as a function of KL and pick a checkpoint near the peak — almost certainly far below 40 nats. (2) Fix the labeler: switch to a different-family judge (kills self-preference), add CoT + order-swap debiasing (kills position bias), and use length-controlled comparisons (kills verbosity/formatting reward). (3) Fix the reward: length-normalize/penalize $r_{\varphi }$; add preference pairs that explicitly penalize confident-agreement-with-wrong-user (anti-sycophancy data); consider WARM or an ensemble with an uncertainty penalty for robustness under the drift. (4) Tighten β so the run can’t reach KL≈40. (5) Longer term, iterated online RLAIF (§8) so the RM stays calibrated on the evolving policy. Meta-lesson: RLAIF removed the human, not the bias — a smoothly rising proxy at KL≈40 was the predictable place to land, not a surprise.

---

10. Reading order

Work through these in order:

1. The AI-feedback blueprint (start here): Constitutional AI: Harmlessness from AI Feedback — Bai et al., 2022 (arXiv:2212.08073) — read the two-phase method carefully: §3 (SL: critique→revise→SFT) and §4 (RL: AI preferences → PM → RLAIF). Note the Goodhart failure they report (boilerplate / over-harsh).
2. The parity evidence: RLAIF vs. RLHF — Lee et al., 2023 (arXiv:2309.00267) — the head-to-head that shows RLAIF ≈ RLHF, the same-size / same-checkpoint labeler result, and direct-RLAIF (d-RLAIF). Read the prompting/debiasing appendix for the CoT + order-swap tricks.
3. The judge, studied: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Zheng et al., 2023 (arXiv:2306.05685) — §3 on position, verbosity, and self-enhancement bias; this is §5 of this lesson in full.
4. Overoptimization, quantified: Scaling Laws for Reward Model Overoptimization — Gao et al., 2022 (arXiv:2210.10760) — the $d=\sqrt{\mathrm{KL}}$ parameterization, the BoN vs RL functional forms, and RM-size/data effects. This is §6.2.
5. A modern hacking mitigation: WARM: On the Benefits of Weight Averaged Reward Models — Ramé et al., 2024 (arXiv:2401.12187) — weight-averaging RMs for ensemble-grade robustness at single-model cost; linear mode connectivity is the key mechanism.
6. RL-free alternatives: Scaling Relationship on Learning Mathematical Reasoning (RFT) — Yuan et al., 2023 (arXiv:2308.01825), then Beyond Human Data (ReST-EM) — Singh et al., 2023 (arXiv:2312.06585) and its predecessor Reinforced Self-Training (ReST) — Gulcehre et al., 2023 (arXiv:2308.08998). Read RFT for the diversity-of-paths insight and ReST-EM for the EM framing and the overfitting-after-few-iterations caveat.
7. (Optional) iterated self-improvement: Self-Rewarding Language Models — Yuan et al., 2024 (arXiv:2401.10020) — RLAIF + iterated DPO fused into one loop (§8).

---

11. What’s next

You’ve now seen the two ways to attack the RLHF cost structure that keep a learned proxy: cheapen the labeler (RLAIF/CAI) and tame the proxy (KL/ensembles/WARM). The next lesson removes the proxy entirely where it can:

1. 09-rl-for-reasoning — RLVR / GRPO: replace the learned RM with a verifier. In math and code, the reward is a checkable fact (unit tests pass, final answer matches), not a learned scalar — so §6’s overoptimization largely dissolves (there’s almost nothing to hack when the reward is ground truth). GRPO further drops the value head, using group-relative sample rewards as the baseline. This is the endpoint of the “make the proxy better → remove the proxy” arc that started in lesson 06. Note the deep link back to §7: RFT/ReST-EM already use verifiable rewards in the filter step — RLVR is what you get when you run that signal through online RL instead of offline SFT.
2. Back to 07-dpo-and-rl-free-preference-optimization — reread DPO now that you’ve seen §8: iterated online DPO is where DPO and the online-collection idea meet, and RLAIF is exactly how you get the labels for each online round without a human. The three ideas (DPO’s RL-free loss, RLAIF’s AI labels, iterated online collection) compose into one cheap, scalable, reasonably-hack-resistant pipeline.
3. Scalable oversight (thread to pull): RLAIF works because judging is easier than generating — but that asymmetry weakens as models approach and exceed human ability on the task. Debate, recursive reward modeling, and weak-to-strong generalization are the frontier attempts to keep AI feedback trustworthy past human-labelable difficulty.

---

12. References

---

Topic hub: rl-for-llms | Reference pages: rlhf, reward-model, kl-regularization-rlhf | Filed: 2026-09-02

Footnotes

1. Lee, Phatale, Mansoor, et al. (Google) — “RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback” (2023), arXiv:2309.00267. ICML 2024. RLAIF ≈ RLHF (and better on harmlessness); same-size/same-checkpoint labeler result; direct-RLAIF (d-RLAIF); CoT + order-swap debiasing. [established] ↩ ↩² ↩³ ↩⁴ ↩⁵

2. Bai, Kadavath, Kundu, et al. (Anthropic) — “Constitutional AI: Harmlessness from AI Feedback” (2022), arXiv:2212.08073. Two-phase (SL critique→revise→SFT; RL=RLAIF) pipeline; harmless-and-non-evasive RL-CAI; observed Goodhart into boilerplate / over-harsh responses. [established] ↩ ↩² ↩³ ↩⁴

3. Zheng, Chiang, Sheng, et al. — “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena” (2023), arXiv:2306.05685. NeurIPS 2023 Datasets & Benchmarks. GPT-4 judge >80% agreement with humans; position, verbosity (“repetitive list attack”), and self-enhancement biases. [established] ↩ ↩² ↩³ ↩⁴

4. Gao, Schulman, Hilton — “Scaling Laws for Reward Model Overoptimization” (2022), arXiv:2210.10760. ICML 2023. The $d=\sqrt{\mathrm{KL}}$ parameterization; BoN vs RL functional forms; larger RM / more data ⇒ higher, later peak; analytic BoN KL $\log n-\frac{n-1}{n}$. [established] ↩ ↩² ↩³

5. Ramé, Vieillard, Hussenot, et al. (Google DeepMind) — “WARM: On the Benefits of Weight Averaged Reward Models” (2024), arXiv:2401.12187. Weight-averaging RMs via linear mode connectivity for ensemble-grade robustness at single-model inference cost. [recent] ↩

6. Yuan, Yuan, Li, et al. — “Scaling Relationship on Learning Mathematical Reasoning with Large Language Models” (Rejection-sampling Fine-Tuning, RFT) (2023), arXiv:2308.01825. Distinct-reasoning-path diversity drives quality; multi-model rejection samples raise LLaMA-7B GSM8K 35.9%→49.3%. [established] ↩ ↩²

7. Singh, Co-Reyes, Agarwal, et al. (Google DeepMind) — “Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models” (ReST-EM) (2023), arXiv:2312.06585. EM framing (Grow/Improve, re-SFT from base); on MATH/APPS with PaLM-2 beats SFT-on-human-data and scales with size; overfits after a few iterations. [established] ↩ ↩²

8. Gulcehre, Paine, Srinivasan, et al. (Google DeepMind) — “Reinforced Self-Training (ReST) for Language Modeling” (2023), arXiv:2308.08998. Grow/Improve with offline RL in the Improve step; introduced on machine translation. [established] ↩

9. Anthony, Tian, Barber — “Thinking Fast and Slow with Deep Learning and Tree Search” (Expert Iteration, ExIt) (2017), arXiv:1705.08439. NeurIPS 2017. Alternates expert improvement (MCTS) with policy distillation (imitation learning); the umbrella for RFT/ReST. [established] ↩

10. Havrilla, Du, Raparthy, et al. (Meta) — “Teaching Large Language Models to Reason with Reinforcement Learning” (2024), arXiv:2403.04642. Expert Iteration nearly as sample-efficient as PPO for LLM reasoning; models rarely explore beyond SFT support. [recent] ↩

11. Yuan, Pang, Cho, Sukhbaatar, Xu, Weston (Meta) — “Self-Rewarding Language Models” (2024), arXiv:2401.10020. The model is its own LLM-as-a-judge to label fresh samples each round, then iterated DPO — RLAIF + iterated DPO fused into one self-improvement loop. [recent] ↩