DPO (Direct Preference Optimization)

Definition

Direct Preference Optimization (Rafailov et al., 2023; arXiv:2305.18290, “Your Language Model is Secretly a Reward Model”) collapses the entire back half of RLHF — reward-model training plus the PPO loop — into a single supervised classification loss on preference pairs. The load-bearing fact: the KL-regularized RLHF objective has a closed-form optimal policy, so the reward is a reparameterization of the policy itself. DPO changes variables from “reward space” to “policy space” before fitting, turning preference learning into maximum likelihood — no RL, no RM, no sampling loop.

Key math

The KL-regularized objective ${\max }_{\pi }{\mathbb{E}}_{y\sim \pi }[r(x,y)]-\beta \mathrm{KL}(\pi \Vert {\pi }_{\text{ref}})$ has the exact optimum (a Boltzmann tilt of the reference, $\beta$ = temperature):
${\pi }^{\ast }(y\mid x)=\frac{1}{Z(x)}{\pi }_{\text{ref}}(y\mid x)\exp (\frac{1}{\beta }r(x,y)),Z(x)={\sum }_y{\pi }_{\text{ref}}(y\mid x)\exp (r/\beta ).$
Invert to read the reward off the policy: $r(x,y)=\beta \log \frac{{\pi }^{\ast }(y\mid x)}{{\pi }_{\text{ref}}(y\mid x)}+\beta \log Z(x)$. Substitute into Bradley–Terry $P(y_w\succ y_l)=\sigma (r_w-r_l)$: the intractable $\beta \log Z(x)$ is response-independent, so it cancels exactly (same prompt $x$). Parameterize ${\pi }^{\ast }\to {\pi }_{\theta }$ and take the NLL:
$\boxed{{\mathcal{L}}_{\text{DPO}}(\theta )=-{\mathbb{E}}_{(x,y_w,y_l)}\left[\log \sigma \right(\beta \log \frac{{\pi }_{\theta }(y_w\mid x)}{{\pi }_{\text{ref}}(y_w\mid x)}-\beta \log \frac{{\pi }_{\theta }(y_l\mid x)}{{\pi }_{\text{ref}}(y_l\mid x)}\left)\right]}$

The implicit reward is ${\hat{r}}_{\theta }(x,y)\equiv \beta \log \frac{{\pi }_{\theta }(y\mid x)}{{\pi }_{\text{ref}}(y\mid x)}$ — DPO is literally the same Bradley–Terry loss as RM training, with the scalar head replaced by the $\beta$-scaled log-ratio. Gradient: raise $\log {\pi }_{\theta }(y_w)$, lower $\log {\pi }_{\theta }(y_l)$, weighted by $\sigma ({\hat{r}}_l-{\hat{r}}_w)$ — large exactly when the pair is currently mis-ordered. $\beta$ is the KL leash as the sigmoid-logit scale (typical $\beta \in [0.05,0.5]$, default $0.1$).

Why it matters

DPO deletes three of RLHF’s four models (RM, critic, and the sampling loop), giving cheap, stable, accessible alignment on the pairs you already collected. But it trades problems: it is offline / off-policy (never sees on-policy samples → distribution shift, the defining limitation), suffers likelihood displacement (both $\log {\pi }_{\theta }(y_w)$ and $\log {\pi }_{\theta }(y_l)$ can fall — Razin et al. 2024, arXiv:2410.08847), overfits near-deterministic preferences, and is sensitive to ${\pi }_{\text{ref}}$/SFT coverage. Monitor absolute chosen-likelihood, not just the margin. The frontier default is iterative/online DPO — resample fresh on-policy pairs each round — which closes most of the gap to PPO. See preference-optimization for the variant family (IPO/KTO/ORPO/SimPO/CPO).

Taught in

• 07-dpo-and-rl-free-preference-optimization — the full derivation (complete-the-KL, $Z(x)$ cancellation), implicit reward, gradient, and failure modes.
• 08-scaling-rlhf-and-alternatives — iterated online DPO and RLAIF-labeled pairs.

See also

• preference-optimization — the umbrella RL-free family
• bradley-terry-model — the preference likelihood DPO reuses
• rlhf — the pipeline DPO compresses
• reward-model — the explicit proxy DPO makes implicit
• kl-regularization-rlhf — the objective DPO solves in closed form
• ppo — the RL loop DPO deletes
• rl-for-llms