PPO (Proximal Policy Optimization)

Definition

Proximal Policy Optimization (Schulman et al., 2017; arXiv:1707.06347) is a first-order policy-gradient method that makes training stable by baking a trust region into the objective: a clipped surrogate that removes the incentive to move the policy too far in a single update. It gets most of TRPO’s near-monotonic-improvement guarantee with only clipping and plain SGD, and enables multiple epochs of reuse per batch via importance sampling. It is the classic RLHF optimizer.

Key math

Probability ratio $r_t(\theta )=\frac{{\pi }_{\theta }(a_t\mid s_t)}{{\pi }_{{\theta }_{\text{old}}}(a_t\mid s_t)}$ (equals 1 at ${\theta }_{\text{old}}$). The clipped surrogate:
$\boxed{L^{\text{CLIP}}(\theta )={\mathbb{E}}_t\left[\min \right(r_t(\theta ){\hat{A}}_t,\mathrm{clip}(r_t(\theta ),1-ϵ,1+ϵ){\hat{A}}_t\left)\right]}$

Both pieces are essential. The clip flattens the objective once the ratio moves a factor $ϵ$ in the helpful direction (zero gradient — stop rewarding overshoot). The min makes it a pessimistic lower bound: when the ratio has moved the wrong way, the unclipped (active-gradient) branch is kept so a corrective gradient still pulls it back. Full objective (with value and entropy terms):
$L_t=L_t^{\text{CLIP}}-c_1(V_{\varphi }(s_t)-{\hat{R}}_t{)}^2+c_2\mathcal{H}[{\pi }_{\theta }(\cdot \mid s_t)]$

Advantages ${\hat{A}}_t$ from GAE, usually normalized per minibatch. Loop: rollout → GAE → $K$ epochs ($\approx 3$–$10$) of minibatch SGD → refresh ${\theta }_{\text{old}}$. Typical $ϵ=0.2$; PPO-Penalty is an adaptive-$\beta$ KL variant, largely superseded by clip.

Why it matters

RLHF is PPO applied to language generation: policy = the LLM, action = next token, reward = reward-model score at EOS minus a per-token KL penalty to a frozen reference. Note the two distinct KLs: the implicit trust-region KL to ${\pi }_{{\theta }_{\text{old}}}$ (enforced by clip, per-step) and the explicit reference KL to ${\pi }_{\text{ref}}$ (in the reward, over the whole run). PPO is operationally heavy — up to four large models (policy, critic, reference, reward) and sensitive to $ϵ$, $\beta$, $K$, and advantage normalization — which motivates GRPO (drop the critic) and DPO (drop the RL loop).

Taught in

• 04-ppo — TRPO → clipped surrogate derivation, the min case analysis, full objective and loop, and the RLHF instantiation.

See also

• trust-region — the principle PPO enforces cheaply
• generalized-advantage-estimation — supplies ${\hat{A}}_t$
• actor-critic — the value head / critic
• kl-regularization-rlhf — the reference-KL in the reward
• reward-model — the reward source
• rlhf — the pipeline PPO sits inside
• rl-for-llms