REINFORCE

Definition

REINFORCE (Williams, 1992) is the simplest concrete policy-gradient algorithm: a Monte Carlo estimator of the policy gradient that samples full trajectories, weights each action’s score by its (reward-to-go) return, and ascends. It is unbiased but high-variance and strictly on-policy — data must be discarded after a single update. Every modern RLHF/RLVR method (including PPO, RLOO, GRPO) is a variance-reduced, sample-reused descendant of this estimator.

Key math

With ${\Psi }_t=$ reward-to-go and $N$ sampled trajectories:
$\hat{g}=\frac{1}{N}{\sum }_{i=1}^N{\sum }_{t=0}^T{\nabla }_{\theta }\log {\pi }_{\theta }(a_t^{(i)}\mid s_t^{(i)})\left({\sum }_{t^{\prime }=t}^T{\gamma }^{t^{\prime }-t}{r}_{t^{\prime }}^{(i)}\right),\theta \leftarrow \theta +\alpha \hat{g}$

Reward-to-go (drop rewards before $t$ — causally uncaused by $a_t$) and a state baseline $b(s_t)$ both lower variance without bias. With $b=V^{\pi }(s_t)$ the weight becomes the advantage. In practice: minimize the surrogate loss with ${\Psi }_t$ detached (stop_gradient):
$L=-\frac{1}{N}{\sum }_i{\sum }_t\log {\pi }_{\theta }(a_t^{(i)}\mid s_t^{(i)}){\Psi }_t^{(i)}$
which is just advantage-weighted maximum likelihood — ordinary cross-entropy on sampled actions, weighted by how good the outcome was.

Why it matters

The surrogate-loss view is exactly how RLHF code looks: generate a completion, score it, and do weighted MLE on the tokens. REINFORCE’s on-policy single-use nature is its central weakness — throwing away expensive LLM rollouts after one step — which is the entire motivation for PPO’s importance-sampling / clipped objective. Conversely, recent LLM methods (RLOO, GRPO) argue PPO’s critic and clipping are often unnecessary and move back toward plain REINFORCE with a good baseline (a per-prompt group mean).

Taught in

• 02-policy-gradients — REINFORCE as the Monte Carlo policy gradient, the surrogate-loss framing, variance and baselines.

See also

• policy-gradient-theorem — the estimator REINFORCE instantiates
• advantage-function — the low-variance weight
• generalized-advantage-estimation — how to estimate that weight well
• ppo — the sample-reusing, trust-region successor
• rlhf — REINFORCE-style weighted MLE on completions
• rl-for-llms