Policy Gradient Theorem

Definition

The Policy Gradient Theorem gives an unbiased, sampleable estimator for the gradient of the RL objective $J(\theta )={\mathbb{E}}_{\tau \sim {\pi }_{\theta }}[R(\tau )]$ with respect to policy parameters — even though the distribution being averaged over depends on those very parameters. It rests on the score-function (log-derivative) trick and, crucially, makes the unknown environment dynamics drop out, so learning is model-free. It is the foundation of REINFORCE, actor-critic, and PPO.

Key math

Score-function trick — turn a gradient of an expectation into an expectation of a gradient:
${\nabla }_{\theta }{\mathbb{E}}_{x\sim p_{\theta }}[f(x)]={\mathbb{E}}_{x\sim p_{\theta }}[f(x){\nabla }_{\theta }\log p_{\theta }(x)]$

Applied with $x=\tau$, $f=R(\tau )$, and the trajectory factorization, every environment term (${\rho }_0$, transitions $P$) is $\theta$-independent and vanishes:
${\nabla }_{\theta }\log p_{\theta }(\tau )={\sum }_t{\nabla }_{\theta }\log {\pi }_{\theta }(a_t\mid s_t)$

giving the general form with a weight ${\Psi }_t$ on each action’s score:
$\boxed{{\nabla }_{\theta }J(\theta )={\mathbb{E}}_{\tau \sim {\pi }_{\theta }}\left[\sum _t{\nabla }_{\theta }\log {\pi }_{\theta }(a_t\mid s_t){\Psi }_t\right]}$

All choices of ${\Psi }_t$ share the same expected gradient but differ in variance: total return $R(\tau )$ (highest) → reward-to-go → baselined reward-to-go → $Q^{\pi }$ → advantage $A^{\pi }$ (lowest). A state-dependent baseline $b(s_t)$ is unbiased because the expected score is zero: ${\mathbb{E}}_{a\sim \pi }[{\nabla }_{\theta }\log {\pi }_{\theta }(a\mid s)]=0$.

Why it matters

Because the gradient flows only through $\log {\pi }_{\theta }$, the reward $R(\tau )$ can be non-differentiable, black-box, even human — exactly the RLHF/RLVR setting where reward is a learned reward model or a pass/fail verifier. For an LLM the autoregressive log-prob sum ${\sum }_t\log p_{\theta }(y_t\mid x,y_{<t})$ is the trajectory log-prob, so the policy gradient reuses the ordinary token log-probs. The huge vocabulary action space is why value-based $\arg \max$ methods fail and policy-gradient methods dominate LLM post-training.

Taught in

• 02-policy-gradients — full derivation: score-function trick, dynamics dropping out, the ${\Psi }_t$ menu, baselines, and the LLM connection.

See also

• reinforce — the Monte Carlo policy-gradient algorithm
• advantage-function — the ideal weight ${\Psi }_t=A^{\pi }$
• actor-critic — learned baseline for the gradient
• ppo — trust-region wrapper around this estimator
• rlhf — where the reward is a learned model
• rl-for-llms