04 · PPO

What you're learning

How to take the policy-gradient machinery from lessons 01–03 and make it stable enough to actually train large policies — including the LLMs behind RLHF. By the end you should be able to derive PPO-Clip from the trust-region idea, explain exactly why the min-of-clipped surrogate does what it does, and map every term onto the RLHF-PPO loop (reward model, KL-to-reference, value head, GAE).

This is the capstone of the foundations tier. Everything downstream — 05-rl-on-token-sequences, GRPO, DPO — is either an application of PPO or a deliberate simplification of it.

---

1. Learning map

graph TD
    A["Policy gradient<br/>(lesson 02)<br/>∇J = E[∇log π · Â]"] --> B["Instability:<br/>one big step wrecks π"]
    A --> C["On-policy staleness:<br/>data dies after 1 update"]
    B --> D["Trust region idea:<br/>bound the policy change"]
    C --> E["Importance sampling ratio<br/>r_t(θ) = π_θ / π_old"]
    D --> F["TRPO<br/>max surrogate s.t. KL ≤ δ<br/>(natural gradient, CG)"]
    E --> F
    F --> G["Too heavy:<br/>2nd-order, Fisher-vector products"]
    G --> H["PPO-Clip<br/>L^CLIP = E[min(rÂ, clip(r)Â)]"]
    E --> H
    F --> I["PPO-Penalty<br/>adaptive-β KL penalty"]
    H --> J["Full PPO loss:<br/>clip + value + entropy"]
    J --> K["Training loop:<br/>rollout → GAE → K epochs SGD"]
    K --> L["RLHF-PPO<br/>policy=LLM, reward=RM − β·KL_ref"]
    L --> M["Foreshadow: DPO (07)<br/>GRPO (09)"]

    style H fill:#4a4,color:#fff
    style L fill:#44a,color:#fff
    style D fill:#a64,color:#fff

Prerequisites (assumed): MDPs and returns (01-mdps-and-the-rl-objective), the policy-gradient theorem and REINFORCE (02-policy-gradients), actor-critic and generalized advantage estimation (03-actor-critic-and-gae).

---

2. Why this matters: the two failure modes of vanilla PG

Recall the policy-gradient estimator from lesson 02, written with an advantage baseline from lesson 03:

${\nabla }_{\theta }J(\theta )={\mathbb{E}}_{(s_t,a_t)\sim {\pi }_{\theta }}\left[{\nabla }_{\theta }\log {\pi }_{\theta }(a_t\mid s_t){\hat{A}}_t\right]$

You estimate this from a batch of rollouts, take a gradient ascent step, throw the data away, and roll out again. This works on toy problems, but it is brutally fragile at scale for two coupled reasons.

Failure mode 1 — a too-large step destroys the policy.
The gradient ${\nabla }_{\theta }J$ is only a local linear approximation of $J$. It tells you the ascent direction, but nothing about how far you can trust it. The mapping from parameter space $\theta$ to policy space ${\pi }_{\theta }$ is wildly non-isometric: a small $\Vert \Delta \theta \Vert$ can produce a huge change in the distribution ${\pi }_{\theta }(\cdot \mid s)$ (and vice versa). If a single SGD step moves $\pi$ too far, the new policy visits a completely different part of the state space, the old advantage estimates become meaningless, and performance can collapse irrecoverably — there is no supervised “ground truth” to snap back to. This is the signature pathology of policy gradient: monotonic-improvement is not guaranteed, and a bad step is often unrecoverable.

Failure mode 2 — on-policy data is single-use.
The expectation above is taken under ${\pi }_{\theta }$ — the current policy. The instant you update $\theta$, your batch was drawn from ${\pi }_{{\theta }_{\text{old}}}\ne {\pi }_{\theta }$, so it is off-policy and the estimator is biased. Vanilla PG therefore does exactly one gradient step per batch of environment interaction. When a rollout is expensive (a robot episode, or — foreshadowing — an LLM generating thousands of tokens against a reward model), throwing the batch away after one step is ruinously sample-inefficient.

The unifying want

Both failures point to the same fix: limit how far the policy is allowed to move per update. If the step is small in policy space (not parameter space), (a) the linear approximation stays valid so improvement is roughly monotone, and (b) ${\pi }_{\theta }\approx {\pi }_{{\theta }_{\text{old}}}$ so we can reuse the batch for several steps before it goes stale. This is the trust region idea, and PPO is the cheap, robust way to enforce it.

---

3. The importance-sampling ratio: reusing off-policy data

To take multiple gradient steps on one batch, we must evaluate the current policy ${\pi }_{\theta }$ on data collected under ${\pi }_{{\theta }_{\text{old}}}$. Importance sampling gives the correction. For any function $f$,

${\mathbb{E}}_{a\sim {\pi }_{\theta }}[f(a)]={\mathbb{E}}_{a\sim {\pi }_{{\theta }_{\text{old}}}}\left[\frac{{\pi }_{\theta }(a\mid s)}{{\pi }_{{\theta }_{\text{old}}}(a\mid s)}f(a)\right].$

Define the probability ratio

$r_t(\theta )=\frac{{\pi }_{\theta }(a_t\mid s_t)}{{\pi }_{{\theta }_{\text{old}}}(a_t\mid s_t)},r_t({\theta }_{\text{old}})=1.$

The surrogate objective — the thing TRPO and PPO both maximize — replaces the log-prob gradient with this ratio:

$L^{\text{CPI}}(\theta )={\mathbb{E}}_t\left[r_t(\theta ){\hat{A}}_t\right]$

(“CPI” = conservative policy iteration.¹) Note ${\nabla }_{\theta }{L}^{\text{CPI}}{|}_{{\theta }_{\text{old}}}={\mathbb{E}}_t[{\nabla }_{\theta }\log {\pi }_{\theta }(a_t\mid s_t){\hat{A}}_t]$ — at $\theta ={\theta }_{\text{old}}$ this is exactly the vanilla policy gradient. So the surrogate is a legitimate stand-in: maximizing it locally is maximizing $J$, but now it is written in a form we can evaluate for $\theta \ne {\theta }_{\text{old}}$ and thus optimize for several epochs.

The surrogate is only trustworthy near ${\theta }_{\text{old}}$

$L^{\text{CPI}}$ is an approximation to the true improvement whose error grows with how far ${\pi }_{\theta }$ drifts from ${\pi }_{{\theta }_{\text{old}}}$. Maximizing it without a constraint is precisely failure mode 1 in disguise: the optimizer will happily drive $r_t$ to enormous values on actions with positive advantage, producing an excessively large policy update. The whole game of TRPO/PPO is how to keep this honest.

The theory behind “how far is too far” is the surrogate-improvement bound: the true return satisfies

$J({\pi }_{\theta })\ge L^{\text{CPI}}(\theta )-C\cdot {\max }_s\mathrm{KL}({\pi }_{{\theta }_{\text{old}}}(\cdot \mid s)\Vert {\pi }_{\theta }(\cdot \mid s)),$

for a constant $C$ depending on the advantage magnitude and $\gamma$.² The penalty term says: trust the surrogate only to the extent the policies stay KL-close. Everything else is a way to operationalize this bound cheaply.

---

4. TRPO: the correct-but-heavy trust region

TRPO takes the bound literally but turns the penalty into a hard constraint (a fixed penalty coefficient $C$ from the theory is far too conservative — steps would be tiny).² It solves, each iteration:

${\max }_{\theta }{\mathbb{E}}_t\left[r_t(\theta ){\hat{A}}_t\right]\text{subject to}\overline{\mathrm{KL}}({\theta }_{\text{old}},\theta )={\mathbb{E}}_t\left[\mathrm{KL}({\pi }_{{\theta }_{\text{old}}}(\cdot \mid s_t)\Vert {\pi }_{\theta }(\cdot \mid s_t))\right]\le \delta .$

This is a constrained optimization in a trust region of radius $\delta$ measured in KL (policy space), not ${\ell }_2$ (parameter space) — which is exactly the fix from §2.

How it’s solved (the heavy part). Linearize the objective and quadratically approximate the constraint around ${\theta }_{\text{old}}$:

${\max }_{\Delta \theta }{g}^{\top }\Delta \theta \text{s.t.}\frac{1}{2}\Delta {\theta }^{\top }F\Delta \theta \le \delta ,$

where $g={\nabla }_{\theta }L$ and $F$ is the Fisher information matrix (the Hessian of the KL, a metric on policy space). The closed-form solution is the natural gradient

$\Delta \theta \propto F^{-1}g,\Delta \theta =\sqrt{\frac{2\delta }{g^{\top }{F}^{-1}g}}{F}^{-1}g.$

You never form $F$ explicitly (it is $|\theta |\times |\theta |$). Instead you compute $F^{-1}g$ with conjugate gradient, using Fisher-vector products $Fv$ obtained via a second backprop through the KL. A backtracking line search then shrinks the step until the KL constraint holds and the surrogate actually improved (guarding against the quadratic approximation being wrong).

TRPO's one-sentence legacy

It proved you can get near-monotonic improvement by constraining the step in KL — but at the cost of conjugate gradient + Fisher-vector products + line search per update, which is second-order, memory-hungry, hard to implement, and awkward to combine with parameter sharing (e.g. a shared actor-critic trunk) or with architectures like dropout/RNNs. PPO’s thesis: get 90% of TRPO’s stability with only first-order SGD.

---

5. PPO-Clip: the main event

PPO throws away the explicit constraint and the second-order machinery. Instead it bakes the trust region directly into the objective so that ordinary first-order SGD cannot move the ratio too far. The clipped surrogate is³:

$\boxed{L^{\text{CLIP}}(\theta )={\mathbb{E}}_t\left[\min (r_t(\theta ){\hat{A}}_t,\mathrm{clip}(r_t(\theta ),1-ϵ,1+ϵ\left){\hat{A}}_t\right)\right]}$

with $\mathrm{clip}(r,1-ϵ,1+ϵ)$ pinning the ratio to $[1-ϵ,1+ϵ]$ (typically $ϵ=0.2$). Two things are happening — the clip and the min — and both are essential. Let’s derive why.

5.1 Case analysis: what clip alone does

Consider one term and the two signs of the advantage.

Positive advantage (${\hat{A}}_t>0$). The action was better than baseline; we want to increase its probability, i.e. push $r_t$ up. The unclipped term $r_t{\hat{A}}_t$ grows without bound as $r_t\to \infty$ — an unlimited incentive to make this action near-certain. The clipped term $\mathrm{clip}(r_t,\cdot ,1+ϵ){\hat{A}}_t$ flattens once $r_t>1+ϵ$: no further reward for pushing the probability higher. Gradient there is zero.

Negative advantage (${\hat{A}}_t<0$). The action was worse than baseline; we want to decrease its probability, pushing $r_t$ down. Now $r_t{\hat{A}}_t$ is negative and becomes more negative as $r_t\to 0$ — an unlimited incentive to crush the probability to zero. The clip floors $r_t$ at $1-ϵ$, so the term flattens once $r_t<1-ϵ$.

In both cases: once the ratio has moved a factor $ϵ$ in the beneficial direction, the objective stops rewarding further movement. That is the trust region — enforced pointwise, per sample, with a clip and no KL computation.

5.2 Why the min? The subtle part

Clipping alone is not enough, and this is the most misunderstood point in PPO. The min takes the smaller (more pessimistic) of the unclipped and clipped terms, making $L^{\text{CLIP}}$ a lower bound (a pessimistic estimate) on the unclipped surrogate. Its job is to handle the case where the ratio moves in the wrong direction.

Suppose ${\hat{A}}_t>0$ but the current update has pushed $r_t<1-ϵ$ (the policy got worse at this good action — maybe because a shared network moved to help other samples). We must retain a gradient to pull it back up. With clip-only, if we clipped here we’d zero the gradient and get stuck. The min resolves it:

${\hat{A}}_t>0:L_t^{\text{CLIP}}=\min (r_t,\mathrm{clip}(r_t)){\hat{A}}_t.$

• If $r_t<1-ϵ$: $\mathrm{clip}(r_t)=1-ϵ>r_t$, and since ${\hat{A}}_t>0$ the smaller product is $r_t{\hat{A}}_t$ (unclipped). Gradient is active — it pulls $r_t$ back toward 1. Good.
• If $r_t>1+ϵ$: clipped value $(1+ϵ)<r_t$, smaller product is $(1+ϵ){\hat{A}}_t$ (clipped, flat). Gradient zero. Good — don’t reward overshoot.

The asymmetry is the whole design. Writing it out fully:

Sign of ${\hat{A}}_t$	Ratio region	Term chosen by min	Gradient?	Effect
${\hat{A}}_t>0$	$r_t<1-ϵ$	unclipped $r_t{\hat{A}}_t$	active	pull ratio back up
${\hat{A}}_t>0$	$1-ϵ\le r_t\le 1+ϵ$	either (equal)	active	normal ascent
${\hat{A}}_t>0$	$r_t>1+ϵ$	clipped, flat	zero	stop rewarding overshoot
${\hat{A}}_t<0$	$r_t>1+ϵ$	unclipped $r_t{\hat{A}}_t$	active	push ratio back down
${\hat{A}}_t<0$	$1-ϵ\le r_t\le 1+ϵ$	either (equal)	active	normal descent
${\hat{A}}_t<0$	$r_t<1-ϵ$	clipped, flat	zero	stop rewarding over-suppression

The clean statement

The clip removes the incentive to move the ratio beyond the trust region in the helpful direction. The min ensures that when the ratio has already left the region in the unhelpful direction, the penalty is not itself clipped away — you keep a corrective gradient. Together: bounded reward for good moves, unbounded corrective pressure for bad ones. That asymmetry is precisely a pessimistic lower bound on improvement.

What clipping does not guarantee

A single term’s gradient being zeroed does not bound the total KL across an update. Clip acts per-sample and per-coordinate; a minibatch can still push the aggregate KL past your intended $\delta$ over $K$ epochs, especially early in training. PPO-Clip is a heuristic trust region, not a certified one — which is why practical PPO monitors KL and often early-stops (see §8). Don’t over-trust $ϵ$ alone.

5.3 PPO-Penalty (adaptive KL) — the other variant

The original PPO paper also proposed a penalty form that stays closer to TRPO’s Lagrangian³:

$L^{\text{KLPEN}}(\theta )={\mathbb{E}}_t\left[r_t(\theta ){\hat{A}}_t-\beta \mathrm{KL}({\pi }_{{\theta }_{\text{old}}}(\cdot \mid s_t)\Vert {\pi }_{\theta }(\cdot \mid s_t))\right],$

with $\beta$ adapted between updates to hit a target KL $d_{\text{targ}}$: if the measured $\overline{\mathrm{KL}}>1.5d_{\text{targ}}$, double $\beta$; if $\overline{\mathrm{KL}}<d_{\text{targ}}/1.5$, halve it. Clip generally matched or beat this and needs no per-run KL target, so PPO-Clip became the default. Keep KLPEN in mind, though — the idea of an adaptive KL coefficient reappears verbatim in RLHF (§7), just against a frozen reference policy rather than ${\pi }_{\text{old}}$.

---

6. The full PPO objective and training loop

Actor-critic (lesson 03) shares structure between the policy and a value function $V_{\varphi }$ (often a shared trunk with two heads). PPO’s total per-step objective combines three terms³:

$L_t(\theta ,\varphi )=\underset{\text{policy}}{\underbrace{L_t^{\text{CLIP}}(\theta )}}-c_1\underset{\text{critic}}{\underbrace{L_t^{\text{VF}}(\varphi )}}+c_2\underset{\text{entropy bonus}}{\underbrace{\mathcal{H}[{\pi }_{\theta }(\cdot \mid s_t)]}}.$

We maximize this (equivalently minimize its negation). Term by term:

Value loss — regress the critic toward the empirical returns (GAE targets from lesson 03, ${\hat{R}}_t={\hat{A}}_t+V_{{\varphi }_{\text{old}}}(s_t)$):

$L_t^{\text{VF}}=(V_{\varphi }(s_t)-{\hat{R}}_t{)}^2.$

Many implementations use a clipped value loss mirroring the policy clip, to stop the critic from moving too far in one update⁴:

$L_t^{\text{VF}}=\max \left[\right(V_{\varphi }-{\hat{R}}_t{)}^2,(\mathrm{clip}(V_{\varphi },V_{{\varphi }_{\text{old}}}-\eta ,V_{{\varphi }_{\text{old}}}+\eta )-{\hat{R}}_t{)}^2].$

Entropy bonus — $\mathcal{H}[{\pi }_{\theta }]=-{\sum }_a{\pi }_{\theta }(a\mid s)\log {\pi }_{\theta }(a\mid s)$; maximizing it keeps the policy stochastic, preventing premature collapse to a deterministic (and possibly wrong) policy early in training. $c_2$ is small (e.g. 0.0–0.01).

Advantage normalization is doing more than you think

Almost every strong PPO implementation normalizes advantages to zero mean / unit variance per minibatch before the clip term. This makes $ϵ$ scale-invariant across tasks and reward magnitudes. Omitting it is one of the most common reasons a “correct” PPO reimplementation silently underperforms — a lesson from the “PPO implementation details” literature (Engstrom et al.; Huang et al.).⁴

The loop

flowchart TD
    A["Initialize policy π_θ,<br/>value V_φ (shared trunk + 2 heads)"] --> B
    B["<b>Collect rollouts</b><br/>run π_θ_old for T steps<br/>store s_t, a_t, r_t,<br/>log π_θ_old(a_t|s_t), V(s_t)"] --> C
    C["<b>Compute advantages</b><br/>GAE(λ): Â_t = Σ (γλ)^l δ_{t+l}<br/>returns R_t = Â_t + V(s_t)<br/>normalize Â_t"] --> D
    D["<b>K epochs of minibatch SGD</b>"] --> E
    subgraph inner ["for K epochs, over shuffled minibatches"]
        E["compute r_t(θ)=π_θ/π_θ_old"] --> F["L = L^CLIP − c1·L^VF + c2·H"]
        F --> G["Adam step on θ, φ"]
        G --> H{"KL(π_old‖π_θ)<br/>> target?"}
        H -- "yes" --> I["early-stop epochs"]
        H -- "no" --> E
    end
    D --> inner
    I --> J["θ_old ← θ"]
    inner --> J
    J --> B

    style B fill:#334,color:#fff
    style C fill:#343,color:#fff
    style D fill:#433,color:#fff

The key structural point vs. vanilla PG: one batch of rollouts is reused for $K$ epochs ($K\approx 3$–$10$) of minibatch updates. The ratio $r_t$ starts at 1 for the whole batch (since $\theta ={\theta }_{\text{old}}$ at the first epoch) and drifts as epochs proceed; the clip keeps that drift bounded. After the $K$ epochs, ${\theta }_{\text{old}}\leftarrow \theta$ and you collect fresh data.

---

7. LLM connection: PPO is the classic RLHF optimizer

This is the bridge to the rest of the curriculum. RLHF (InstructGPT⁵, and the early GPT-/Claude-style alignment pipelines) is PPO applied to language generation, where the “environment” is degenerate — it’s just the token sequence itself. Map the RL abstractions onto an LLM:

RL concept	RLHF instantiation
Policy ${\pi }_{\theta }$	The LLM being fine-tuned (init from the SFT model)
State $s_t$	Prompt + tokens generated so far $(x,y_{<t})$
Action $a_t$	The next token $y_t$ from the vocabulary
Episode / trajectory	One full generated response $y_{1:T}$
Reward	Reward-model score at EOS, minus a per-token KL penalty (below)
Value $V_{\varphi }$	A value head on the LLM (lesson 03), predicting return-to-go per token
Advantage ${\hat{A}}_t$	GAE over the per-token rewards (lesson 03)

The reward: RM score minus KL-to-reference

The reward model $r_{\psi }$ (trained separately on human preference comparisons) outputs a scalar for the whole response, available only at the final token. If you optimized that alone, PPO would find degenerate high-reward gibberish — reward hacking / over-optimization of an imperfect $r_{\psi }$. The fix is a per-token KL penalty to a frozen reference policy ${\pi }_{\text{ref}}$ (the SFT model, held fixed). The per-token reward is:

$r_t=\underset{\text{RM score, terminal only}}{\underbrace{r_{\psi }(x,y_{1:T})1[t=T]}}-\underset{\text{per-token KL penalty}}{\underbrace{\beta (\log {\pi }_{\theta }(y_t\mid x,y_{<t})-\log {\pi }_{\text{ref}}(y_t\mid x,y_{<t}))}}.$

The bracketed term is a single-sample estimate of $\mathrm{KL}({\pi }_{\theta }\Vert {\pi }_{\text{ref}})$ at position $t$. So the total objective PPO maximizes is:

${\mathbb{E}}_{x\sim \mathcal{D},y\sim {\pi }_{\theta }}[r_{\psi }(x,y)-\beta \mathrm{KL}({\pi }_{\theta }(\cdot \mid x)\Vert {\pi }_{\text{ref}}(\cdot \mid x)\left)\right].$

Two different KLs — don't conflate them

There are two KL terms in RLHF-PPO, and they do different jobs:

1. The PPO trust-region KL — between ${\pi }_{\theta }$ and ${\pi }_{{\theta }_{\text{old}}}$, enforced implicitly by the clip. Keeps each optimization step stable. Changes every update (${\theta }_{\text{old}}$ is refreshed).
2. The RLHF reference KL — between ${\pi }_{\theta }$ and the frozen ${\pi }_{\text{ref}}$, added explicitly into the reward with coefficient $\beta$. Keeps the model from drifting away from fluent, sensible language over the whole run and anchors it against reward-model exploitation. ${\pi }_{\text{ref}}$ never moves.

Why the reference KL is load-bearing: the reward model is a learned, imperfect proxy for human preference. Its errors are largest exactly in the regions of output space the SFT model rarely visits — so an unconstrained optimizer marches straight into $r_{\psi }$‘s blind spots and produces text that scores high but is degenerate. The $\beta \mathrm{KL}({\pi }_{\theta }\Vert {\pi }_{\text{ref}})$ term is a leash: it makes drifting into those under-trained regions costly, trading a little reward for staying in the distribution where $r_{\psi }$ is trustworthy. Tuning $\beta$ is the central reward–fluency knob of the whole pipeline; some pipelines adapt it exactly like PPO-Penalty’s adaptive-$\beta$ scheme from §5.3.

Why RLHF-PPO is operationally painful

A full RLHF-PPO setup keeps four models in play: the policy (trainable), the value/critic head (trainable, often a second full copy the size of the policy), the frozen reference ${\pi }_{\text{ref}}$, and the frozen reward model $r_{\psi }$.⁵ That’s roughly 2× the memory of the policy just for the critic and reference, plus a separate RM. It’s also notoriously sensitive to $\beta$, the KL controller, advantage normalization, and reward-scaling. This operational weight is exactly what the next methods attack.

Foreshadowing: why GRPO and DPO exist

• 09-rl-for-reasoning (critic-free) — removes the value head/GAE entirely. GRPO samples a group of responses per prompt and uses their mean/std reward as the baseline (a Monte-Carlo advantage), keeping the PPO-style clipped ratio objective but deleting the second trainable model.⁶ Motivation: kill the critic’s memory and instability.
• 07-dpo-and-rl-free-preference-optimization (RL-free) — removes PPO, the reward model, and the sampling loop. DPO shows the RLHF objective (reward + reference-KL) has a closed-form optimum, letting you fit the policy directly on preference pairs with a simple classification-style loss.⁷ Motivation: skip online RL altogether when you have preference data.

Both are best understood as deliberate deletions from the PPO recipe you now hold in full — which is why PPO is the foundations capstone.

---

8. Hyperparameters and failure modes

The PPO pitfalls checklist

• $ϵ$ (clip) too large → effectively no trust region; you’re back to unstable vanilla PG and can collapse. Too small → glacial learning. Default $0.2$; RLHF often uses smaller.
• $K$ epochs too high → by the last epoch ${\pi }_{\theta }$ has drifted far from ${\pi }_{{\theta }_{\text{old}}}$, most samples are clipped (zero gradient) and the surviving importance weights are high-variance and stale. Symptom: KL spikes, entropy craters, reward collapses. Fix: fewer epochs (3–4) or KL early-stopping.
• KL blowup → the policy KL (to ${\pi }_{\text{old}}$ or, in RLHF, to ${\pi }_{\text{ref}}$) runs away in a single update. Because clip is only a heuristic bound, always monitor KL and either early-stop the epoch loop when $\mathrm{KL}>1.5d_{\text{targ}}$ or raise the penalty $\beta$.
• No advantage normalization → $ϵ$ becomes task/reward-scale-dependent; training is fragile and non-portable (§6).
• Value clipping / value-loss scale ($c_1$) → an over-eager critic destabilizes the shared trunk and corrupts the advantages the actor depends on; clip the value update and keep $c_1\approx 0.5$. In RLHF, a common choice is to not share the trunk (separate value model) precisely to decouple this.
• Reward/return scaling → unnormalized rewards make the advantage magnitudes (and thus the effective learning rate through ${\hat{A}}_t$) unpredictable; use running return normalization.
• Entropy collapse → without the entropy bonus (or with too small $c_2$) the policy goes deterministic too early and stops exploring. In RLHF this shows up as mode-collapse / repetitive outputs.

Typical starting points (continuous control / Atari-scale): $ϵ=0.2$, $K=3\text{–}10$, GAE $\gamma =0.99,\lambda =0.95$, $c_1=0.5$, $c_2\in [0,0.01]$, Adam LR $\approx 3\times 10^{-4}$, minibatch 32–256, rollout length 2048/env.³ RLHF differs: much smaller LR ($\sim 10^{-6}$), $K=1$–$4$, small $ϵ$, and the dominant knob is $\beta$ (reference-KL), tuned to a target KL of a few nats over the response.⁵

---

9. Quizzes

Quiz 1: Why does $L^{\text{CLIP}}$ use min rather than just clipping the ratio?

Answer

Clipping alone would zero the gradient whenever $r_t$ leaves $[1-ϵ,1+ϵ]$ in either direction — including when the ratio has moved the wrong way and we need a corrective gradient to pull it back. The min makes the objective a pessimistic lower bound: it selects the clipped (flat) branch only when the ratio has moved in the beneficial direction beyond the trust region (so we stop rewarding overshoot), but keeps the unclipped (active-gradient) branch when the ratio has moved in the harmful direction (so correction still happens). Concretely, for ${\hat{A}}_t>0$ with $r_t<1-ϵ$: $\min (r_t,1-ϵ){\hat{A}}_t=r_t{\hat{A}}_t$, which has a live gradient pulling $r_t$ back up. Clip-without-min would get stuck. So: bounded reward for good moves, unbounded corrective pressure for bad ones.

Quiz 2: In RLHF-PPO there are two KL terms. Name both, say which is implicit vs. explicit, and what each protects against.

Answer

(1) Trust-region KL, $\mathrm{KL}({\pi }_{\theta }\Vert {\pi }_{{\theta }_{\text{old}}})$ — implicit, enforced by the PPO clip (never computed as a loss term, only monitored). Protects against a single optimization step being too large, i.e. failure mode 1. ${\theta }_{\text{old}}$ is refreshed each iteration.
(2) Reference KL, $\mathrm{KL}({\pi }_{\theta }\Vert {\pi }_{\text{ref}})$ — explicit, subtracted from the reward with coefficient $\beta$ against a frozen reference (the SFT model). Protects against the policy drifting out of the distribution where the reward model is reliable — i.e. reward hacking / over-optimization and loss of fluency — over the entire run. They are easy to conflate because both are KLs bounding policy movement, but one bounds step size (short horizon, moving anchor) and the other bounds total drift (long horizon, fixed anchor).

Quiz 3: You run PPO with $K=15$ epochs per batch and reward suddenly collapses mid-training while measured KL spikes. Mechanistically, what happened?

Answer

With too many epochs, ${\pi }_{\theta }$ drifts far from the ${\pi }_{{\theta }_{\text{old}}}$ that generated the batch. Two things compound: (a) most samples now have $r_t$ outside $[1-ϵ,1+ϵ]$ in the beneficial direction, so they’re clipped and contribute zero gradient — you’re training on an ever-shrinking fraction of the batch; (b) the samples that do have live gradients carry large, high-variance importance corrections computed against a now-stale behavior policy, so updates get noisy and biased. The aggregate KL exceeds the intended trust region (clip is only a per-sample heuristic, not a global bound), the policy lands in a region where the old advantages are meaningless, and performance collapses. Fixes: drop $K$ to 3–4, add KL early-stopping, and/or lower the LR.

Quiz 4: TRPO computes a natural-gradient step $\Delta \theta \propto F^{-1}g$. What is $F$, why not just use $g$ (the vanilla gradient), and why did PPO abandon this?

Answer

$F$ is the Fisher information matrix — the local second-order (Hessian) approximation of the KL divergence between ${\pi }_{\theta }$ and ${\pi }_{\theta +\Delta \theta }$. It defines a metric on policy space. The vanilla gradient $g$ is steepest ascent in parameter (${\ell }_2$) space, but parameter distance is a poor proxy for policy distance — the same $\Vert \Delta \theta \Vert$ can barely move $\pi$ or wreck it. Preconditioning by $F^{-1}$ makes the step steepest-ascent in KL geometry, so a fixed trust-region radius $\delta$ corresponds to a fixed change in the policy distribution regardless of curvature. PPO abandoned it because forming/inverting $F$ (or even doing conjugate-gradient Fisher-vector products plus a line search) every update is expensive, second-order, memory-heavy, and hard to fuse with shared actor-critic trunks and stochastic architectures. PPO’s bet: emulate the trust region with a first-order clipped objective and plain SGD — cheaper, and empirically about as stable.

---

10. Practice problems

Problem 1: Compute the clipped objective and its gradient by hand

Setup. $ϵ=0.2$. For a single sample the old policy assigned probability ${\pi }_{{\theta }_{\text{old}}}(a\mid s)=0.40$. Consider two candidate current values: (i) ${\pi }_{\theta }=0.60$ with $\hat{A}=+2$; (ii) ${\pi }_{\theta }=0.60$ with $\hat{A}=-2$. For each, compute $r$, the unclipped term, the clipped term, $L^{\text{CLIP}}$, and state whether the gradient w.r.t. $\theta$ is active or zero.

Solution.
$r=0.60/0.40=1.5$ in both cases; the clip range is $[0.8,1.2]$, so $\mathrm{clip}(1.5)=1.2$.

(i) $\hat{A}=+2$ (good action, ratio pushed up past the region):

• unclipped: $r\hat{A}=1.5\times 2=3.0$
• clipped: $1.2\times 2=2.4$
• $L^{\text{CLIP}}=\min (3.0,2.4)=2.4$ → clipped branch chosen → gradient zero. Interpretation: the action is already $1.5\times$ more likely; we’ve overshot the trust region in the helpful direction, so PPO stops rewarding further increase. Correct behavior.

(ii) $\hat{A}=-2$ (bad action, but ratio moved the wrong way — up):

• unclipped: $r\hat{A}=1.5\times (-2)=-3.0$
• clipped: $1.2\times (-2)=-2.4$
• $L^{\text{CLIP}}=\min (-3.0,-2.4)=-3.0$ → unclipped branch chosen → gradient active. Interpretation: we made a bad action more likely; the min keeps the full penalty (unclipped) so the gradient strongly pushes $r$ back down. This is exactly the case the min (not clip alone) exists to handle.

Problem 2: Set $\beta$ from a target reference-KL

Setup. You’re running RLHF-PPO. You want the fine-tuned policy to stay within an average $\mathrm{KL}({\pi }_{\theta }\Vert {\pi }_{\text{ref}})\approx 6$ nats per response. You start with $\beta =0.1$. After an epoch you measure the average per-response reference-KL at $10$ nats and the mean RM score is still rising. Using PPO-Penalty’s adaptive rule (target $d_{\text{targ}}=6$), what do you do, and what’s the trade-off you’re managing?

Solution.
Measured KL $=10>1.5\times d_{\text{targ}}=9$, so the adaptive rule fires: increase $\beta$ (e.g. double it to $0.2$). Larger $\beta$ makes deviation from ${\pi }_{\text{ref}}$ more costly, pulling the effective per-response KL back toward the $6$-nat target on the next iteration.
The trade-off: $\beta$ is the reward-vs-fidelity dial. Too small → the policy chases the reward model into its blind spots (reward hacking: high RM score, degenerate/dishonest text), and fluency degrades. Too large → the policy is leashed so tightly to the SFT model that it barely improves on the reward and RLHF gains vanish. The fact that RM score is still rising while KL is only mildly over target suggests a modest $\beta$ increase (not a large one) — you want to rein in drift without freezing learning. This adaptive-$\beta$ control loop is the RLHF analogue of TRPO’s/PPO-Penalty’s KL controller from §5.3, applied to the frozen reference instead of ${\pi }_{\text{old}}$.

---

11. Reading order

Work through these in order:

1. Primary source (start here): Proximal Policy Optimization Algorithms — Schulman et al., 2017 (arXiv:1707.06347) — short and readable; §3 (clipped surrogate) and §5 (the loss with value + entropy) are the core. Note Figure 1’s per-sample plot of the clipped objective vs. $r$ — it is §5.2 in a picture.
2. The trust-region backstory: Trust Region Policy Optimization — Schulman et al., 2015 (arXiv:1502.05477) — read for the surrogate-improvement bound and the natural-gradient/CG machinery PPO is simplifying. You can skim the proofs; internalize the constrained objective.
3. Best pedagogical treatment + code: OpenAI Spinning Up — PPO — clean derivation of the clip intuition and a minimal reference implementation; do the “you should be able to…” exercises.
4. The RLHF instantiation: Training language models to follow instructions with human feedback (InstructGPT) — Ouyang et al., 2022 (arXiv:2203.02155) — read §3.5 and Appendix for the exact RM-score-minus-KL reward and the value-head/PPO setup; this is §7 of this lesson in the wild.
5. Production reference: HuggingFace TRL — PPOTrainer docs — see the concrete four-model setup (policy, ref, reward, value), KL controller, and the config knobs from §8. The best bridge from theory to a real training run.
6. (Optional but worth it): “The 37 Implementation Details of PPO” (Huang et al.) and “Implementation Matters in Deep RL” (Engstrom et al.) — why advantage normalization, value clipping, and reward scaling matter more than the headline algorithm.

---

12. What’s next

Two threads, in order:

1. 05-rl-on-token-sequences — makes the §7 mapping rigorous: how an autoregressive LM is an MDP, per-token vs. sequence-level rewards, credit assignment across a response, and the exact plumbing of value heads and GAE over tokens. This is where PPO becomes RLHF for real.
2. Circle back to 03-actor-critic-and-gae — reread the GAE section now that you’ve seen the value head and per-token advantages in action; the $\lambda$ knob and the bias–variance trade-off will land differently.

Then the two “deletions from PPO”: 09-rl-for-reasoning (drop the critic) and 07-dpo-and-rl-free-preference-optimization (drop the RL loop entirely).

---

References

---

Topic hub: rl-for-llms | Reference page: ppo | Filed: 2026-09-02

Footnotes

1. Kakade & Langford (2002). “Approximately Optimal Approximate Reinforcement Learning.” ICML 2002, pp. 267–274. Origin of Conservative Policy Iteration (CPI) and the $L^{\text{CPI}}$ surrogate objective that TRPO and PPO build on. ↩

2. Schulman, Levine, Abbeel, Jordan, Moritz (2015). “Trust Region Policy Optimization.” arXiv:1502.05477. Source for the surrogate-improvement bound, the KL-constrained trust-region objective, and the natural-gradient / conjugate-gradient / Fisher-vector-product solution. ↩ ↩²

3. Schulman, Wolski, Dhariwal, Radford, Klimov (2017). “Proximal Policy Optimization Algorithms.” arXiv:1707.06347. Source for the clipped surrogate $L^{\text{CLIP}}$ (§3), the adaptive-KL penalty variant (§5), the combined clip+value+entropy objective (§5), and the continuous-control/Atari default hyperparameters (§3 uses $ϵ=0.2$). ↩ ↩² ↩³ ↩⁴

4. Engstrom et al. (2020), “Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO,” arXiv:2005.12729; and Huang et al. (2022), “The 37 Implementation Details of Proximal Policy Optimization,” ICLR Blog Track (https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/). Document value-loss clipping and per-minibatch advantage normalization as code-level optimizations that account for much of PPO’s practical performance. ↩ ↩²

5. Ouyang et al. (2022). “Training language models to follow instructions with human feedback” (InstructGPT). arXiv:2203.02155. The canonical RLHF-PPO instantiation: LLM policy initialized from SFT, per-token reward = terminal RM score minus KL-to-frozen-reference penalty, value head, and the policy/critic/reward/reference model setup; also the RLHF-side hyperparameter regime. ↩ ↩² ↩³

6. Shao et al. (2024). “DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models.” arXiv:2402.03300. Introduces GRPO (Group Relative Policy Optimization): removes the value network, using the standardized group-mean reward as the baseline while keeping the PPO-style clipped ratio objective. [recent] ↩

7. Rafailov et al. (2023). “Direct Preference Optimization: Your Language Model is Secretly a Reward Model.” arXiv:2305.18290. Shows the RLHF reward-plus-reference-KL objective has a closed-form optimal policy, enabling direct fitting on preference pairs with a classification-style loss (no reward model, no sampling loop). [recent] ↩