Generalized Advantage Estimation (GAE)

Definition

Generalized Advantage Estimation (Schulman et al., 2015; arXiv:1506.02438) estimates the advantage $A^{\pi }(s,a)$ as an exponentially-weighted average of $k$-step advantage estimators, giving a single knob $\lambda \in [0,1]$ that continuously trades bias against variance. It is the advantage estimator inside PPO and RLHF-PPO, converting a sparse terminal reward into per-token advantages via a one-pass backward recursion over critic-based TD errors.

Key math

The TD error is the 1-step advantage estimate (the atom of GAE):
${\delta }_t=r_t+\gamma V_{\varphi }(s_{t+1})-V_{\varphi }(s_t)$

The $k$-step estimator telescopes into discounted TD errors, ${\hat{A}}_t^{(k)}={\sum }_{l=0}^{k-1}{\gamma }^l{\delta }_{t+l}$. Averaging over all $k$ with weights $(1-\lambda ){\lambda }^{k-1}$ collapses to:
$\boxed{{\hat{A}}_t^{\text{GAE}(\gamma ,\lambda )}=\sum _{l=0}^{\infty }(\gamma \lambda {)}^l{\delta }_{t+l}}{\hat{A}}_t={\delta }_t+(\gamma \lambda ){\hat{A}}_{t+1}$

$\lambda$ is the bias–variance knob (distinct from $\gamma$, which sets objective far-sightedness):

• $\lambda \to 0$: ${\hat{A}}_t\to {\delta }_t$ — pure 1-step TD. Low variance, high bias (trusts $V_{\varphi }$).
• $\lambda \to 1$: ${\hat{A}}_t\to R_t-V_{\varphi }(s_t)$ — pure Monte Carlo advantage. High variance, low bias.

The paired value target is $V_t^{\text{targ}}={\hat{A}}_t^{\text{GAE}}+V_{\varphi }(s_t)$ (detached). Typical: $\gamma \in [0.99,0.999]$, $\lambda \in [0.9,0.97]$.

Why it matters

GAE is what lets a sequence-level reward propagate credit back to individual tokens in RLHF-PPO: the terminal reward-model score plus a dense per-token KL penalty become a token-level reward stream, which the $\gamma \lambda$-discounted backward pass turns into token-level advantages. Setting $\lambda$ too high (near 1) on long sparse-reward generations reintroduces Monte Carlo variance; lowering it leans on the critic’s bootstrap to shorten the effective horizon. Critic-free methods (GRPO) drop GAE entirely in favor of a group baseline.

Taught in

• 03-actor-critic-and-gae — full derivation from the TD error, the $\lambda$-average collapse, and worked backward-recursion examples.
• 04-ppo — GAE as the advantage estimator feeding the clipped surrogate.

See also

• advantage-function — the quantity GAE estimates
• value-function — the critic $V_{\varphi }$ supplying the TD errors
• actor-critic — the loop GAE lives in
• ppo — consumes ${\hat{A}}_t^{\text{GAE}}$
• rl-for-llms