Advantage Function

Definition

The advantage function $A^{\pi }(s,a)$ answers the single most important question in policy-gradient RL: was action $a$ better or worse than the policy’s average behavior in state $s$? It is the action-value minus the state-value, a re-centered signal whose sign says “do more / do less” and whose expectation under $\pi$ is zero. It is the ideal weight ${\Psi }_t$ in the policy gradient and the target that GAE estimates.

Key math

$A^{\pi }(s,a)=Q^{\pi }(s,a)-V^{\pi }(s)$

Centered by construction — subtracting $V^{\pi }$ (a state-only baseline) recenters $Q^{\pi }$:
${\mathbb{E}}_{a\sim \pi (\cdot \mid s)}[A^{\pi }(s,a)]={\sum }_a\pi (a\mid s)[Q^{\pi }(s,a)-V^{\pi }(s)]=V^{\pi }(s)-V^{\pi }(s)=0$

This zero-mean property is exactly why using $V^{\pi }$ as a baseline is unbiased (it contributes $V^{\pi }(s){\mathbb{E}}_a[\nabla \log \pi ]=0$) while cutting variance. In the PG theorem the advantage is the lowest-variance choice of weight:
${\nabla }_{\theta }J(\theta )=\mathbb{E}\left[{\sum }_t{\nabla }_{\theta }\log {\pi }_{\theta }(a_t\mid s_t)A^{\pi }(s_t,a_t)\right]$

Estimated in practice via the TD error (${\hat{A}}_t^{(1)}={\delta }_t=r_t+\gamma V_{\varphi }(s_{t+1})-V_{\varphi }(s_t)$) or its $\lambda$-weighted sum. Advantages are usually normalized (zero mean, unit variance) per batch before the actor update.

Why it matters

For LLMs the reward is a single sequence-level scalar (a reward-model or verifier score), so the entire credit-assignment problem is: how do we distribute that one number across all the token-actions that produced it? The advantage is the answer — per-token $A_t$ tell the model which tokens to uplift or suppress. In RLHF-PPO it is computed by GAE over token-level rewards; in GRPO it is estimated group-relatively, $A^{(i)}=(R^{(i)}-\mathrm{mean})/\mathrm{std}$, with no critic at all.

Taught in

• 01-mdps-and-the-rl-objective — definition and the zero-mean baseline property.
• 02-policy-gradients — why $A^{\pi }$ is the ideal, minimum-variance weight ${\Psi }_t$.
• 03-actor-critic-and-gae — estimating $A^{\pi }$ well with a critic and GAE.

See also

• value-function — $A^{\pi }=Q^{\pi }-V^{\pi }$
• generalized-advantage-estimation — the practical estimator
• policy-gradient-theorem — where the advantage is used
• actor-critic — learns $V_{\varphi }$ to form $A$
• rl-for-llms