Value Function

Definition

A value function measures expected future return under a policy $\pi$, making the RL objective tractable at the level of individual states and actions. The state-value $V^{\pi }(s)$ scores a state; the action-value $Q^{\pi }(s,a)$ scores committing to an action first, then following $\pi$. Their difference is the advantage $A^{\pi }(s,a)=Q^{\pi }(s,a)-V^{\pi }(s)$. A learned approximation $V_{\varphi }$ is the critic in actor-critic methods.

Key math

Defined on an MDP via the return $G_t$:
$V^{\pi }(s)={\mathbb{E}}_{\tau \sim \pi }[G_t\mid s_t=s],Q^{\pi }(s,a)={\mathbb{E}}_{\tau \sim \pi }[G_t\mid s_t=s,a_t=a]$

Linked by averaging over the policy’s action choice:
$V^{\pi }(s)={\mathbb{E}}_{a\sim \pi (\cdot \mid s)}[Q^{\pi }(s,a)]={\sum }_a\pi (a\mid s)Q^{\pi }(s,a)$

Bellman expectation equations (a consequence of $G_t=r_t+\gamma G_{t+1}$ plus Markov):
$V^{\pi }(s)={\sum }_a\pi (a\mid s){\sum }_{s^{\prime }}P(s^{\prime }\mid s,a)[R(s,a)+\gamma V^{\pi }(s^{\prime })]$
$Q^{\pi }(s,a)=R(s,a)+\gamma {\sum }_{s^{\prime }}P(s^{\prime }\mid s,a){\sum }_{a^{\prime }}\pi (a^{\prime }\mid s^{\prime })Q^{\pi }(s^{\prime },a^{\prime })$

These are linear in $V$/$Q$ (fixed policy). The Bellman optimality equations replace ${\sum }_a\pi$ with ${\max }_a$ (nonlinear) and characterize $V^{\ast },Q^{\ast }$. The recursion is what makes values estimable: you can bootstrap $V^{\pi }(s)$ from $V^{\pi }(s^{\prime })$ (temporal-difference learning) instead of waiting for full returns.

Why it matters

$V^{\pi }(s)$ is the near-optimal baseline in policy-gradient methods: subtracting it centers the learning signal at zero, cutting gradient variance without adding bias, and turning $Q^{\pi }$ into the advantage. In RLHF-PPO the critic is a value head on the transformer trunk predicting per-token return-to-go — one of up to four large networks in memory (policy, critic, reward, reference). Because a per-token critic over long sparse-reward sequences is expensive and noisy, critic-free methods (GRPO, RLOO) replace it with a group/batch baseline.

Taught in

• 01-mdps-and-the-rl-objective — $V^{\pi },Q^{\pi },A^{\pi }$ and the Bellman equations, from first principles.
• 03-actor-critic-and-gae — learning $V_{\varphi }$ as a critic and training it by regression.

See also

• advantage-function — $A^{\pi }=Q^{\pi }-V^{\pi }$, the centered signal
• actor-critic — using a learned $V_{\varphi }$ as baseline + bootstrap
• generalized-advantage-estimation — TD errors and $\lambda$-averaging over $V_{\varphi }$
• markov-decision-process — the setting these are defined on
• rl-for-llms