03 · Actor-Critic & GAE

What you're learning

How to turn the policy gradient of lessons 01–02 into a low-variance, online algorithm by learning a value function alongside the policy — and how Generalized Advantage Estimation gives you a single knob $\lambda$ to trade bias against variance in the advantage estimate. By the end you should be able to derive ${\hat{A}}_t^{\text{GAE}(\gamma ,\lambda )}$ from the TD error and compute it by hand for a short trajectory.

---

1. Learning map

graph TD
    A["Policy gradient<br/>∇J = E[∇log π · A]<br/>(lesson 02)"] --> B["Advantage A_t<br/>needs an estimate"]
    B --> C["Monte Carlo return<br/>unbiased, high variance"]
    B --> D["Bootstrapped / TD<br/>biased, low variance"]
    C --> E["Bias–variance tradeoff"]
    D --> E
    D --> F["Learn a critic V_φ<br/>= actor-critic"]
    F --> G["TD error δ_t<br/>= 1-step advantage"]
    G --> H["n-step returns<br/>interpolate MC ↔ TD"]
    H --> I["GAE(γ,λ)<br/>exponential avg of n-step"]
    E --> I
    F --> J["Critic loss<br/>regress V_φ → returns"]
    I --> K["Actor-critic update loop"]
    J --> K
    K --> L["RLHF-PPO<br/>value head + token-level GAE"]
    L --> M["Critic-free methods<br/>GRPO — lesson 09"]

    style F fill:#4a4,color:#fff
    style I fill:#44a,color:#fff
    style L fill:#a44,color:#fff

Path: advantage needs estimating → MC vs TD are the two extremes → learn a critic to bootstrap → TD error is the 1-step case → n-step interpolates → GAE is the right exponential interpolation → train the critic → assemble the loop → apply to LLMs.

---

2. Why this matters

Lesson 02 left us with the policy gradient in its advantage form:

${\nabla }_{\theta }J(\theta )={\mathbb{E}}_{\tau \sim {\pi }_{\theta }}\left[{\sum }_t{\nabla }_{\theta }\log {\pi }_{\theta }(a_t\mid s_t)A^{\pi }(s_t,a_t)\right]$

The whole game is now the quality of your estimate ${\hat{A}}_t$ of the true advantage $A^{\pi }(s_t,a_t)=Q^{\pi }(s_t,a_t)-V^{\pi }(s_t)$. Everything else is fixed.

If you estimate $A_t$ with the Monte Carlo return $R_t={\sum }_{l\ge 0}{\gamma }^l{r}_{t+l}$ minus a baseline (REINFORCE-with-baseline), the estimate is unbiased but its variance grows with the horizon: a single reward at $t+500$ perturbs the credit assigned to every earlier action. Gradients become noisy, learning is slow and sample-hungry.

The actor-critic insight: learn a value function $V_{\varphi }$ and use it to bootstrap — replace far-future sampled rewards with the critic’s prediction. This slashes variance (you stop summing hundreds of noisy reward terms) at the cost of some bias (the critic is imperfect). GAE then makes the bias-variance tradeoff continuous and controllable with one hyperparameter.

The one-sentence version

Actor-critic = “use a learned $V_{\varphi }$ to shorten the horizon you have to sample over”; GAE = “don’t commit to one horizon — take an exponentially-weighted average over all of them.”

---

3. The actor-critic idea

Two learned components, trained jointly¹:

• Actor ${\pi }_{\theta }(a\mid s)$ — the policy. Updated by policy gradient.
• Critic $V_{\varphi }(s)$ — a learned estimate of the state-value $V^{\pi }(s)={\mathbb{E}}_{\pi }[{\sum }_{l\ge 0}{\gamma }^l{r}_{t+l}\mid s_t=s]$. Updated by regression.

The critic serves two jobs at once:

1. Baseline — subtracting $V_{\varphi }(s_t)$ from the return reduces variance without introducing bias (any function of $s_t$ alone is a valid baseline, since ${\mathbb{E}}_{a\sim \pi }[{\nabla }_{\theta }\log {\pi }_{\theta }(a\mid s)b(s)]=0$).
2. Bootstrap — $V_{\varphi }(s_{t+1})$ lets us truncate the sampled return and substitute a prediction for the tail, which is where the variance reduction beyond a plain baseline comes from.

Baseline vs bootstrap are different mechanisms

Using $V_{\varphi }$ purely as a baseline ($R_t-V_{\varphi }(s_t)$) keeps the estimate unbiased but still has MC-level variance in $R_t$. The extra variance reduction of actor-critic comes from bootstrapping — replacing the sampled tail ${\sum }_{l\ge k}{\gamma }^l{r}_{t+l}$ with ${\gamma }^k{V}_{\varphi }(s_{t+k})$. That substitution is what introduces bias. Don’t conflate the two roles.

Why bootstrapping reduces variance

The MC return $R_t={\sum }_{l\ge 0}{\gamma }^l{r}_{t+l}$ is a sum over many random rewards; each future reward’s stochasticity (from policy, transitions, reward noise) adds to $\mathrm{Var}[R_t]$. Bootstrapping after $k$ steps,

$R_t^{(k)}={\sum }_{l=0}^{k-1}{\gamma }^l{r}_{t+l}+{\gamma }^k{V}_{\varphi }(s_{t+k}),$

replaces the infinite noisy tail with a single (nearly) deterministic function evaluation. Fewer random terms → lower variance. But $V_{\varphi }\ne V^{\pi }$ in general, so you inherit the critic’s approximation error as bias.

---

4. The bias-variance tradeoff in advantage estimation

Define the $k$-step advantage estimator:

${\hat{A}}_t^{(k)}=\underset{k\text{-step return }{R}_t^{(k)}}{\underbrace{\sum _{l=0}^{k-1}{\gamma }^l{r}_{t+l}+{\gamma }^k{V}_{\varphi }(s_{t+k})}}-V_{\varphi }(s_t).$

The two endpoints:

Estimator	$k$	Bias	Variance
1-step TD	$k=1$	high (leans on $V_{\varphi }$ immediately)	low
$n$-step	$1<k<\infty$	medium	medium
Monte Carlo	$k\to \infty$	zero (if episodic, $V_{\varphi }$ term vanishes)	high

Intuition: the more real rewards you sum before bootstrapping, the closer to unbiased (you trust the sampled signal, not the critic) but the noisier (more random terms).

Quiz: Why is the $k\to \infty$ (Monte Carlo) estimator unbiased regardless of how bad $V_{\varphi }$ is?

Answer

As $k\to \infty$ in an episodic task, the bootstrap term ${\gamma }^k{V}_{\varphi }(s_{t+k})\to 0$ (either ${\gamma }^k\to 0$ or the episode terminates with $V=0$). The estimator collapses to $R_t-V_{\varphi }(s_t)$. The subtracted $V_{\varphi }(s_t)$ is a pure baseline — a function of $s_t$ only — so it drops out of the expected gradient without adding bias. All the critic-dependence that could bias the estimate lives in the bootstrap term, which has vanished. Hence unbiased. The price is that $R_t$ retains full MC variance.

---

5. The TD error as a 1-step advantage

The temporal-difference error is:

$\boxed{{\delta }_t=r_t+\gamma V_{\varphi }(s_{t+1})-V_{\varphi }(s_t)}$

This is exactly ${\hat{A}}_t^{(1)}$: the 1-step return $r_t+\gamma V_{\varphi }(s_{t+1})$ minus the current-state value baseline $V_{\varphi }(s_t)$.

Key fact: if $V_{\varphi }=V^{\pi }$ (the true value function), then ${\delta }_t$ is an unbiased estimator of the advantage $A^{\pi }(s_t,a_t)$:

${\mathbb{E}}_{s_{t+1},r_t}[{\delta }_t\mid s_t,a_t]=\underset{=Q^{\pi }(s_t,a_t)}{\underbrace{\mathbb{E}[r_t+\gamma V^{\pi }(s_{t+1})\mid s_t,a_t]}}-V^{\pi }(s_t)=A^{\pi }(s_t,a_t).$

In practice $V_{\varphi }\ne V^{\pi }$, so ${\delta }_t$ is biased — this is the “TD is biased” statement made precise. It’s low-variance because it involves only one sampled reward.

${\delta }_t$ is the atom

Everything in GAE is built out of TD errors. The $k$-step advantage telescopes into a sum of discounted TD errors:
${\hat{A}}_t^{(k)}={\sum }_{l=0}^{k-1}{\gamma }^l{\delta }_{t+l}.$
Check the $k=2$ case by hand: ${\delta }_t+\gamma {\delta }_{t+1}=[r_t+\gamma V(s_{t+1})-V(s_t)]+\gamma [r_{t+1}+\gamma V(s_{t+2})-V(s_{t+1})]=r_t+\gamma r_{t+1}+{\gamma }^{2}V(s_{t+2})-V(s_t)$. The intermediate $V(s_{t+1})$ terms cancel — a telescoping sum.

---

6. Generalized Advantage Estimation (GAE)

We have a whole family $\{{\hat{A}}_t^{(k)}{\}}_{k=1}^{\infty }$ trading bias for variance. Which $k$ do you pick? Schulman et al. (2015, arXiv:1506.02438) answer²: don’t pick — take an exponentially-weighted average over all $k$, with decay $\lambda \in [0,1]$.

Define GAE as the $\lambda$-weighted average of the $k$-step estimators:

${\hat{A}}_t^{\text{GAE}(\gamma ,\lambda )}=(1-\lambda )\left({\hat{A}}_t^{(1)}+\lambda {\hat{A}}_t^{(2)}+{\lambda }^2{\hat{A}}_t^{(3)}+\cdots \right)=(1-\lambda ){\sum }_{k=1}^{\infty }{\lambda }^{k-1}{\hat{A}}_t^{(k)}.$

The $(1-\lambda )$ normalizer makes the weights $(1-\lambda ){\lambda }^{k-1}$ sum to 1. Now substitute the telescoped form ${\hat{A}}_t^{(k)}={\sum }_{l=0}^{k-1}{\gamma }^l{\delta }_{t+l}$ and swap the order of summation. Each TD error ${\delta }_{t+l}$ appears in every ${\hat{A}}_t^{(k)}$ with $k>l$, so its total weight is $(1-\lambda ){\gamma }^l{\sum }_{k=l+1}^{\infty }{\lambda }^{k-1}=(1-\lambda ){\gamma }^l\cdot \frac{{\lambda }^l}{1-\lambda }=(\gamma \lambda {)}^l$. This collapses beautifully:

$\boxed{{\hat{A}}_t^{\text{GAE}(\gamma ,\lambda )}=\sum _{l=0}^{\infty }(\gamma \lambda {)}^l{\delta }_{t+l}}$

A discounted sum of future TD errors — with the same exponential form as the return itself, but discount $\gamma \lambda$ instead of $\gamma$. It computes in one backward pass:

${\hat{A}}_t={\delta }_t+(\gamma \lambda ){\hat{A}}_{t+1}.$

$\lambda$ is the bias-variance knob

• $\lambda \to 0$: ${\hat{A}}_t^{\text{GAE}}\to {\delta }_t$ — pure 1-step TD. Low variance, high bias (fully trusts $V_{\varphi }$).
• $\lambda \to 1$: ${\hat{A}}_t^{\text{GAE}}\to {\sum }_{l\ge 0}{\gamma }^l{\delta }_{t+l}=R_t-V_{\varphi }(s_t)$ (the full telescope) — pure Monte Carlo advantage. High variance, low bias.
• $0<\lambda <1$: smooth interpolation.

$\gamma$ and $\lambda$ play distinct roles: $\gamma$ controls how far-sighted the objective is (and adds bias by downweighting distant true rewards); $\lambda$ controls how much you trust the critic vs. sampled rewards for a fixed objective. Typical values: $\gamma \in [0.99,0.999]$, $\lambda \in [0.9,0.97]$.²

$\gamma \lambda$ is not "just a smaller discount"

It’s tempting to see $\sum (\gamma \lambda {)}^l{\delta }_{t+l}$ and think you’ve merely lowered the discount. But the summand is the TD error $\delta$, not a reward $r$. Each ${\delta }_{t+l}$ already contains a bootstrap $V_{\varphi }(s_{t+l+1})$. Lowering $\lambda$ downweights distant TD corrections, i.e. leans harder on the earliest bootstrap — that’s a bias choice, not a myopia choice.

Quiz: What does $\lambda$ control, and what happens to bias and variance at $\lambda =0$ and $\lambda =1$?

Answer

$\lambda$ is the bias-variance tradeoff knob for the advantage estimate — it sets the exponential decay rate on how much weight future TD errors receive, i.e. how far you sample real rewards before leaning on the critic’s bootstrap.

• $\lambda =0$: ${\hat{A}}_t={\delta }_t$, the 1-step TD estimate. Minimum variance, maximum bias (relies entirely on $V_{\varphi }$ after one step).
• $\lambda =1$: ${\hat{A}}_t={\sum }_l{\gamma }^l{\delta }_{t+l}=R_t-V_{\varphi }(s_t)$, the Monte Carlo advantage. Maximum variance, minimum bias (only $V_{\varphi }(s_t)$ enters, as a pure baseline).
  In between you interpolate. Note this bias is w.r.t. the $\gamma$-discounted objective; $\gamma \ne 1$ already introduces its own bias relative to the undiscounted return.

Quiz: Derive why GAE at $\lambda =1$ equals the Monte Carlo advantage $R_t-V_{\varphi }(s_t)$.

Answer

Set $\lambda =1$: ${\hat{A}}_t={\sum }_{l\ge 0}{\gamma }^l{\delta }_{t+l}$. Expand and telescope:
${\sum }_{l\ge 0}{\gamma }^l(r_{t+l}+\gamma V(s_{t+l+1})-V(s_{t+l})).$
Group the value terms: $+{\gamma }^l\cdot \gamma V(s_{t+l+1})={\gamma }^{l+1}V(s_{t+l+1})$ from term $l$, and $-{\gamma }^{l+1}V(s_{t+l+1})$ from term $l+1$. They cancel pairwise for all $l\ge 0$, leaving only $-{\gamma }^{0}V(s_t)=-V(s_t)$ from the first term. The reward terms sum to ${\sum }_{l\ge 0}{\gamma }^l{r}_{t+l}=R_t$. Result: ${\hat{A}}_t=R_t-V(s_t)$. ∎

---

7. Training the critic

The actor uses ${\hat{A}}_t$; the critic must be trained so those advantages are meaningful. Two common targets:

Target 1 — bootstrapped $\lambda$-return (consistent with GAE). The value target that pairs naturally with GAE is

$V_t^{\text{targ}}={\hat{A}}_t^{\text{GAE}}+V_{\varphi }(s_t),$

so the critic regresses toward its own advantage-corrected estimate. Loss:

$L^V(\varphi )={\mathbb{E}}_t\left[\frac{1}{2}(V_{\varphi }(s_t)-V_t^{\text{targ}}{)}^2\right].$

Target 2 — Monte Carlo returns. Simpler: regress $V_{\varphi }(s_t)$ directly onto the empirical return $R_t$. Unbiased target but high-variance, so slower critic learning.

Stop-gradient on the value target

$V_t^{\text{targ}}$ is computed from the old critic (and old GAE) and treated as a constant — you do not backprop through it. Otherwise you get a moving-target instability where the network chases its own bootstrap. In practice the targets (“returns”) are computed once per rollout batch and detached.

PPO's clipped value loss

RLHF-PPO usually clips the value update the same way it clips the policy³, to stop the critic from moving too far in one step:
$L^V={\mathbb{E}}_t\left[\max \right((V_{\varphi }-V^{\text{targ}}{)}^2,(\text{clip}(V_{\varphi },V_{{\varphi }_{\text{old}}}\pm ϵ)-V^{\text{targ}}{)}^2\left)\right].$
This is a heuristic; some implementations drop it.⁴ Flag when reading code which variant is used.

---

8. Putting it together: the actor-critic loop

graph TD
    A["Rollout: run π_θ<br/>collect (s_t, a_t, r_t)"] --> B["Critic forward:<br/>V_φ(s_t) for all t"]
    B --> C["Compute TD errors<br/>δ_t = r_t + γV(s_{t+1}) − V(s_t)"]
    C --> D["GAE backward pass<br/>Â_t = δ_t + γλ·Â_{t+1}"]
    D --> E["Value targets<br/>V_t^targ = Â_t + V_φ(s_t)"]
    D --> F["Actor loss<br/>−E[log π_θ(a_t|s_t)·Â_t]<br/>(Â_t detached)"]
    E --> G["Critic loss<br/>½E[(V_φ(s_t) − V_t^targ)²]"]
    F --> H["Backprop → update θ (and φ)"]
    G --> H
    H --> A

    style D fill:#44a,color:#fff
    style F fill:#4a4,color:#fff
    style G fill:#a44,color:#fff

Per iteration:

1. Collect a rollout with the current policy.
2. Evaluate $V_{\varphi }$ at every visited state.
3. Compute ${\delta }_t$, then ${\hat{A}}_t$ by the backward recursion.
4. Update the actor with ${\nabla }_{\theta }\mathbb{E}[\log {\pi }_{\theta }(a_t\mid s_t){\hat{A}}_t]$ (advantages detached — they’re targets, not differentiated through).
5. Update the critic by regressing $V_{\varphi }(s_t)\to V_t^{\text{targ}}$.

Advantages are usually normalized (subtract mean, divide by std over the batch) before the actor update — a variance-reduction trick that stabilizes the gradient scale.⁴

Shared vs separate networks

Actor and critic can share a trunk with two heads (common at LLM scale, where the trunk is the transformer) or be fully separate. Sharing saves compute but couples the two losses — you need a coefficient $c_v$ on the value loss to balance them: $L=L^{\pi }+c_v{L}^V-c_e\mathcal{H}[\pi ]$ (the last term is an entropy bonus for exploration).

---

9. LLM connection: GAE in RLHF-PPO

This is where actor-critic hits frontier practice. In classic RLHF-PPO (InstructGPT-style)⁵:

• Actor = the LLM policy ${\pi }_{\theta }$, generating tokens autoregressively. Each token is an action $a_t$; the state $s_t$ is the prompt + tokens generated so far.
• Critic = a value head — usually a linear layer on top of the (shared or separately-copied) transformer backbone — outputting a scalar $V_{\varphi }(s_t)$ per token position.
• Reward is sparse: the reward model scores the whole completion, giving a single scalar at the final token. To this, a per-token KL penalty against the reference policy is added at every position:
  $r_t=-\beta (\log {\pi }_{\theta }(a_t\mid s_t)-\log {\pi }_{\text{ref}}(a_t\mid s_t))+\underset{\text{only at final token}}{\underbrace{r_{\text{RM}}\cdot 1[t=T]}}.$

GAE then converts this token-level reward-plus-penalty stream into token-level advantages ${\hat{A}}_t$ via the exact same backward recursion. This is what lets the sequence-level reward propagate credit back to individual tokens: the terminal RM reward flows backward through $\gamma \lambda$-discounted TD errors, while the dense KL penalty shapes every step.

Why the value head is expensive at LLM scale

The critic is (roughly) a second model the size of the policy. Even as a value head on a shared trunk, PPO typically keeps a separate frozen reference model and a reward model in memory too — up to four large networks (policy, critic, reward, reference) during training.⁵ The critic also needs its own warmup and can be hard to fit well (value estimation over long token sequences with sparse reward is noisy). This memory + tuning burden is a major practical pain point.

Teaser — critic-free methods (lesson 09)

The cost of the value network motivates critic-free RLHF. GRPO (Group Relative Policy Optimization) drops $V_{\varphi }$ entirely: instead of a learned baseline, it samples a group of $G$ completions per prompt and uses the group’s mean reward as the baseline, with advantages ${\hat{A}}_i=(r_i-\text{mean}(\{r_j\}))/\text{std}(\{r_j\})$. No critic, no value head, no GAE — just Monte Carlo returns with a batch-computed baseline.⁶ We’ll dissect the tradeoffs in 09-rl-for-reasoning.

---

10. Practice problems

Practice problem 1: Compute GAE for a 3-step trajectory

Problem: A trajectory has three transitions. Rewards $r_0=1,r_1=0,r_2=2$. Critic values $V(s_0)=0.5,V(s_1)=1.0,V(s_2)=1.5,V(s_3)=0$ (terminal). Use $\gamma =1.0$, $\lambda =0.5$. Compute ${\delta }_0,{\delta }_1,{\delta }_2$ and then ${\hat{A}}_0^{\text{GAE}}$.

Worked solution:
TD errors ($\gamma =1$, so ${\delta }_t=r_t+V(s_{t+1})-V(s_t)$):

• ${\delta }_2=2+V(s_3)-V(s_2)=2+0-1.5=0.5$
• ${\delta }_1=0+V(s_2)-V(s_1)=0+1.5-1.0=0.5$
• ${\delta }_0=1+V(s_1)-V(s_0)=1+1.0-0.5=1.5$

Backward recursion ${\hat{A}}_t={\delta }_t+\gamma \lambda {\hat{A}}_{t+1}$ with $\gamma \lambda =0.5$:

• ${\hat{A}}_2={\delta }_2=0.5$
• ${\hat{A}}_1={\delta }_1+0.5\cdot {\hat{A}}_2=0.5+0.25=0.75$
• ${\hat{A}}_0={\delta }_0+0.5\cdot {\hat{A}}_1=1.5+0.375=1.875$

Check via the direct sum: ${\hat{A}}_0={\sum }_l(\gamma \lambda {)}^l{\delta }_l=1.5+0.5(0.5)+0.25(0.5)=1.5+0.25+0.125=1.875$. ✓

Sanity: at $\lambda =1$ this would be $R_0-V(s_0)=(1+0+2)-0.5=2.5$ (pure MC advantage); at $\lambda =0$ it would be ${\delta }_0=1.5$ (pure TD). Our $\lambda =0.5$ answer $1.875$ sits between — as it must.

Practice problem 2: Diagnose a variance blow-up

Problem: You’re training PPO on a 1000-token generation task with sparse terminal reward. You set $\gamma =1.0,\lambda =1.0$. Training is unstable with huge gradient variance. Explain mechanistically why, and what single knob you’d turn first.

Worked solution:
At $\lambda =1$, ${\hat{A}}_t=R_t-V_{\varphi }(s_t)$ — pure Monte Carlo. With a sparse terminal reward and a 1000-step horizon, every token’s advantage depends on the entire future reward stream (the single terminal RM score plus 1000 KL terms). The estimator is unbiased but its variance scales with the horizon: one noisy completion perturbs the credit assigned to all 1000 tokens identically. The critic $V_{\varphi }$ is only being used as a per-token baseline, not to shorten the effective horizon — so you get none of actor-critic’s variance reduction.

First knob: lower $\lambda$ (e.g. to $0.95$). This reintroduces bootstrapping: distant TD errors get downweighted by $(\gamma \lambda {)}^l$, so each token’s advantage leans on the (lower-variance) critic prediction for the far future instead of summing 1000 noisy terms. You accept a little bias from the imperfect $V_{\varphi }$ in exchange for a large variance reduction. (Secondary: normalize advantages, and consider $\gamma <1$.)

---

11. Reading order

Work through these in order:

1. Primary source: High-Dimensional Continuous Control Using Generalized Advantage Estimation — Schulman et al. 2015 (arXiv:1506.02438) — the GAE paper. Read Sections 2–3 for the derivation; the ${\hat{A}}_t=\sum (\gamma \lambda {)}^l{\delta }_{t+l}$ collapse is Eq. (16).
2. Textbook grounding: Sutton & Barto, Reinforcement Learning: An Introduction (2nd ed.), Ch. 13.5 (actor-critic methods) and Ch. 12 (eligibility traces / $\lambda$-returns — GAE is the policy-gradient analogue of TD($\lambda$)).
3. Implementation view: OpenAI Spinning Up — Vanilla Policy Gradient & the GAE-Lambda advantage — read the pseudocode and the compute_gae / discounted-cumsum trick.
4. LLM application: the InstructGPT paper (Ouyang et al. 2022) and any modern PPO-RLHF codebase (e.g. TRL’s PPOTrainer) — trace how the value head and token-level GAE are wired.

---

12. What’s next

1. 04-ppo — GAE is the advantage estimator inside PPO; next we add the clipped surrogate objective and trust-region intuition that make the actor update stable.
2. 02-policy-gradients — revisit if the advantage form of the gradient or the baseline-doesn’t-bias argument felt shaky; GAE only makes sense on that foundation.
3. 09-rl-for-reasoning (teaser above) — how dropping the critic entirely and using a group baseline trades the GAE machinery for sampling cost.

---

References

---

Reference topic: rl-for-llms | Filed: 2026-09-02

Footnotes

1. Sutton & Barto, Reinforcement Learning: An Introduction (2nd ed., 2018). Ch. 13.5 (actor–critic methods) for the actor/critic decomposition and baseline argument; Ch. 12 (eligibility traces / TD($\lambda$) and $\lambda$-returns), of which GAE is the policy-gradient analogue. ↩

2. Schulman, Moritz, Levine, Jordan, Abbeel (2015). “High-Dimensional Continuous Control Using Generalized Advantage Estimation.” arXiv:1506.02438. GAE origin; the ${\hat{A}}_t={\sum }_l(\gamma \lambda {)}^l{\delta }_{t+l}$ collapse is Eq. (16), and the reported experiments use $\gamma \in [0.99,0.995]$, $\lambda \in [0.9,0.99]$. ↩ ↩²

3. Schulman, Wolski, Dhariwal, Radford, Klimov (2017). “Proximal Policy Optimization Algorithms.” arXiv:1707.06347. ↩

4. Engstrom et al. (2020), “Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO,” arXiv:2005.12729; and Huang et al. (2022), “The 37 Implementation Details of Proximal Policy Optimization,” ICLR Blog Track (https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/). Cover the value-loss clipping heuristic and per-minibatch advantage normalization as code-level details that materially affect PPO performance. ↩ ↩²

5. Ouyang et al. (2022). “Training language models to follow instructions with human feedback” (InstructGPT). arXiv:2203.02155. The canonical RLHF-PPO setup: LLM policy + value head, per-token KL-to-reference penalty, and the policy/critic/reward/reference model configuration. ↩ ↩²

6. Shao et al. (2024). “DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models.” arXiv:2402.03300. Introduces GRPO (Group Relative Policy Optimization): drops the value function and uses the standardized group mean reward as the baseline. [recent] ↩