Actor-Critic

Definition

Actor-critic methods learn two components jointly: an actor ${\pi }_{\theta }$ (the policy, updated by policy gradient) and a critic $V_{\varphi }$ (a learned state-value estimate, updated by regression). The critic serves two distinct roles — a baseline (variance reduction) and a bootstrap (replacing the sampled reward tail with a prediction) — turning the high-variance Monte Carlo REINFORCE estimator into a lower-variance, more online algorithm. It is the structural backbone of PPO and RLHF-PPO.

How it works

The actor uses the advantage form of the gradient with an estimated ${\hat{A}}_t$ (typically GAE):
${\nabla }_{\theta }J=\mathbb{E}\left[{\sum }_t{\nabla }_{\theta }\log {\pi }_{\theta }(a_t\mid s_t){\hat{A}}_t\right]$

Two roles of $V_{\varphi }$, don’t conflate:

• Baseline — subtracting $V_{\varphi }(s_t)$ is unbiased (any function of $s_t$ alone), but leaves MC-level variance in the return.
• Bootstrap — replacing the sampled tail with ${\gamma }^k{V}_{\varphi }(s_{t+k})$ cuts variance further, at the cost of the critic’s approximation error as bias.

The critic is trained by regression (detached target):
$L^V(\varphi )={\mathbb{E}}_t[\frac{1}{2}(V_{\varphi }(s_t)-V_t^{\text{targ}}{)}^2],V_t^{\text{targ}}={\hat{A}}_t^{\text{GAE}}+V_{\varphi }(s_t)$

Combined objective with a value coefficient and entropy bonus: $L=L^{\pi }+c_v{L}^V-c_e\mathcal{H}[\pi ]$. Actor and critic may share a trunk with two heads (common at LLM scale) or be fully separate.

Why it matters

In RLHF-PPO the actor is the LLM and the critic is a value head on the transformer backbone predicting per-token return-to-go. It is roughly a second model the size of the policy; together with a frozen reference and reward model, PPO keeps up to four large networks in memory. The critic is also hard to fit well over long, sparse-reward token sequences. This memory-and-tuning burden is precisely what critic-free methods (GRPO, RLOO) attack by replacing the learned $V_{\varphi }$ with a group/batch baseline.

Taught in

• 03-actor-critic-and-gae — the actor-critic idea, baseline vs bootstrap, training the critic, the full loop, and the RLHF value head.
• 04-ppo — actor-critic structure inside the full PPO objective.

See also

• value-function — the critic $V_{\varphi }$
• generalized-advantage-estimation — how the actor’s ${\hat{A}}_t$ is formed
• policy-gradient-theorem — the actor update
• ppo — actor-critic + trust region
• rl-for-llms