07 · DPO & RL-Free Preference Optimization

What you're learning

How the entire RLHF back half — reward model plus the PPO loop — collapses into a single supervised classification loss on preference pairs. The load-bearing fact: the KL-regularized RLHF objective has a closed-form optimal policy, so the reward is redundant with the policy itself. You’ll derive that optimum, invert it to read the reward off the policy, substitute into Bradley-Terry to watch the partition function cancel, and land on the DPO loss. Then: what its gradient does, where it breaks (likelihood displacement, offline distribution shift, the on-policy debate), and the zoo of variants (IPO, KTO, ORPO, SimPO, CPO) — each a targeted patch on one crack.

This is Tier 2, lesson 07. It assumes the full RLHF pipeline from 06-rlhf-pipeline — the Bradley-Terry reward model, the SFT→RM→PPO staging, and the $β KL (π_{θ} ∥ π_{ref})$ leash — and the token-level MDP + reference-KL plumbing from 05-rl-on-token-sequences. Lesson 06 built the four-model machine; this lesson deletes three of the four models.

1. Learning map

graph TD
    A["RLHF pipeline (06):<br/>RM + PPO + KL, four models"] --> B["Observation: the RL objective<br/>max E[r] − β·KL(π‖π_ref)<br/>has a KNOWN optimum"]
    B --> C["Derive π*(y|x) =<br/>(1/Z) π_ref exp(r/β)"]
    C --> D["Invert: r(x,y) =<br/>β log(π*/π_ref) + β log Z"]
    D --> E["Substitute into Bradley-Terry<br/>P(y_w≻y_l)=σ(r_w−r_l)"]
    E --> F["Z(x) CANCELS<br/>(same x, appears in both terms)"]
    F --> G["DPO loss:<br/>−log σ(β Δlog-ratio)<br/>supervised, on pairs"]
    G --> H["'Implicit reward'<br/>r̂_θ = β log(π_θ/π_ref)"]
    G --> I["Gradient: up y_w, down y_l,<br/>weighted by how wrong<br/>the implicit ordering is"]
    G --> J["Failure modes"]
    J --> J1["Offline / off-policy<br/>distribution shift"]
    J --> J2["Likelihood displacement<br/>(both logp's can drop)"]
    J --> J3["Overfit deterministic prefs<br/>→ IPO"]
    J --> J4["Reference sensitivity /<br/>SFT coverage"]
    J1 --> K["Iterative / online DPO"]
    J3 --> V["Variants:<br/>IPO · KTO · ORPO · SimPO · CPO"]
    J4 --> V

    style B fill:#44a,color:#fff
    style F fill:#a44,color:#fff
    style G fill:#4a4,color:#fff
    style J2 fill:#a64,color:#fff

Prerequisites (assumed): the Bradley-Terry preference model and RM loss (06-rlhf-pipeline §6), the KL-to-frozen-reference reward shaping and the reward-vs-KL trade-off (05-rl-on-token-sequences §5, kl-regularization-rlhf), and sequence log-probs as sums of per-token log-probs (05-rl-on-token-sequences §4).

2. Why this matters: the reward model was never the point

Recall the operational weight of RLHF-PPO from lesson 06: four models resident at once (trainable policy + value head, frozen reference, frozen reward model), a generation-bound online loop, and notorious sensitivity to $β$ , advantage normalization, and reward scaling. And recall its defining pathology: the reward model $r_{ϕ}$ is a learned proxy, PPO is an adversary, and the policy drifts into the RM’s blind spots (reward overoptimization / Goodhart, §8 of lesson 06).

Now ask the subversive question. In the RLHF pipeline we (1) fit a reward model to preferences, then (2) optimize a policy to maximize that reward under a KL leash. But step (2) is a known optimization problem with a closed-form answer. If we can write the optimal policy as an explicit function of the reward, maybe we can also invert it — write the reward as an explicit function of the optimal policy — and skip ever materializing $r_{ϕ}$ at all. If the reward is just a reparameterization of the policy, then the policy is secretly its own reward model. (That is literally the DPO paper’s subtitle: Your Language Model is Secretly a Reward Model, arXiv:2305.18290.)[

If that works, the entire back half of RLHF becomes a supervised loss on the preference pairs you already collected for the reward model — no separate RM to train, no sampling from the policy during training, no value head, no PPO clip, no online adversary to hold at bay with a KL budget. One model being trained, one frozen reference for log-ratios, and a binary-cross-entropy loss. That is the DPO bet.

The one-sentence thesis

The KL-regularized reward-maximization objective and the policy are two coordinates for the same object; DPO changes variables from “reward space” to “policy space” before fitting, so preference learning becomes direct maximum likelihood on the policy — no RL, no RM, no sampling loop.

Here is the contrast we are about to earn, in one picture:

flowchart TD
    subgraph PPO["RLHF-PPO (lesson 06) — online RL loop"]
        direction TB
        A1["Preference pairs (x, y_w, y_l)"] --> A2["Train reward model r_φ<br/>(Bradley-Terry loss)"]
        A2 --> A3["Freeze r_φ"]
        A3 --> A4["LOOP: sample y ~ π_θ"]
        A4 --> A5["Score r_φ(x,y) at EOS<br/>− β·KL to π_ref per token"]
        A5 --> A6["GAE + value head V_φ"]
        A6 --> A7["PPO-clip update π_θ"]
        A7 -->|"refresh, repeat"| A4
        A7 --> A8["Aligned policy"]
    end
    subgraph DPO["DPO (this lesson) — one supervised loss"]
        direction TB
        B1["Preference pairs (x, y_w, y_l)"] --> B2["Compute log π_θ, log π_ref<br/>on y_w and y_l (forward passes)"]
        B2 --> B3["L = −log σ(β·[Δ_w − Δ_l])<br/>Δ = log(π_θ/π_ref)"]
        B3 --> B4["Backprop, SGD on π_θ"]
        B4 --> B5["Aligned policy"]
    end

    style A4 fill:#a44,color:#fff
    style A2 fill:#a64,color:#fff
    style B3 fill:#4a4,color:#fff

Four models and an online sampling loop on the left; two forward passes and a classification loss on the right. The rest of this lesson is (a) why the right column is a valid stand-in for the left, and (b) exactly when it isn’t.

3. The centerpiece: deriving the closed-form optimal policy

Everything hinges on one classical result. We derive it slowly because every later step is an algebraic consequence.

3.1 The objective

The RLHF objective from lesson 05/06 — reward maximization with a KL leash to a frozen reference — is, per prompt $x$ and with the KL written out:

$max_{π} E_{x \sim D} E_{y \sim π (\cdot ∣ x)} [r (x, y)] - β KL (π (\cdot ∣ x) ∥ π_{ref} (\cdot ∣ x)) .$

The expectation over $x$ decouples the problem: we can solve it independently for each prompt $x$ , because $π (\cdot ∣ x)$ for different $x$ share no constraint other than each being a distribution. So fix $x$ and expand the KL as an expectation:

$max_{π (\cdot ∣ x)} E_{y \sim π} [r (x, y)] - β E_{y \sim π} [lo g \frac{π ( y ∣ x )}{π _{ref} ( y ∣ x )}] .$

Crucially, this is a variational problem over the function $π (\cdot ∣ x)$ — a distribution over the (astronomically large but finite) space of responses $y$ — with the single constraint $\sum_{y} π (y ∣ x) = 1$ . We are not parameterizing by $θ$ yet; we ask what the unconstrained optimal distribution is.

3.2 Complete the square into a single KL

Divide the objective by $β > 0$ (doesn’t change the argmax) and flip the sign to make it a minimization:

$min_{π} E_{y \sim π} [lo g \frac{π ( y ∣ x )}{π _{ref} ( y ∣ x )} - \frac{1}{β} r (x, y)] .$

Now the trick: fold the reward term inside the logarithm by writing $\frac{1}{β} r = lo g exp (r / β)$ :

$= min_{π} E_{y \sim π} [lo g \frac{π ( y ∣ x )}{π _{ref} ( y ∣ x ) e x p ( \frac{1}{β} r ( x , y ) )}] .$

The denominator is almost a probability distribution over $y$ — it’s a non-negative function of $y$ — but it doesn’t normalize to 1. Fix that by dividing and multiplying by its normalizer. Define the partition function

$Z (x) \equiv \sum_{y} π_{ref} (y ∣ x) exp (\frac{1}{β} r (x, y)),$

which depends on $x$ (and on $r$ and $π_{ref}$ ) but not on $π$ — it’s a constant with respect to the thing we’re optimizing. And define the distribution

$π^{*} (y ∣ x) \equiv \frac{1}{Z ( x )} π_{ref} (y ∣ x) exp (\frac{1}{β} r (x, y))$

which is a valid distribution by construction ( $\geq 0$ , sums to 1 because $Z$ normalizes it). Substitute $π_{ref} exp (r / β) = Z (x) π^{*}$ into the objective:

$min_{π} E_{y \sim π} [lo g \frac{π ( y ∣ x )}{Z ( x ) π ^{*} ( y ∣ x )}] = min_{π} = KL (π ∥ π^{*}) E_{y \sim π} [lo g \frac{π ( y ∣ x )}{π ^{*} ( y ∣ x )}] - const in π lo g Z (x) .$

3.3 Read off the optimum

We’ve rewritten the whole regularized objective as $KL (π (\cdot ∣ x) ∥ π^{*} (\cdot ∣ x)) - lo g Z (x)$ . The second term doesn’t depend on $π$ . The first term is a KL divergence, which is $\geq 0$ and equals $0$ iff $π = π^{*}$ . Therefore the minimizer is immediate:

$π^{*} (y ∣ x) = \frac{1}{Z ( x )} π_{ref} (y ∣ x) exp (\frac{1}{β} r (x, y))$

This is the exact, closed-form optimal policy of the KL-regularized RLHF objective — no approximation, no RL. It’s a Boltzmann / Gibbs tilt of the reference: take $π_{ref}$ , reweight each response by $exp (r / β)$ , renormalize. Two sanity checks that make it memorable:

As $β \to \infty$ (infinite KL penalty), $exp (r / β) \to 1$ for all $y$ , so $π^{*} \to π_{ref}$ : crank the leash tight and you never leave the reference.
As $β \to 0$ (no penalty), $exp (r / β)$ becomes a hard max — $π^{*}$ collapses onto $ar g max_{y} r (x, y)$ : no leash and you go to the single highest-reward response.

$β$ is literally the temperature of this Boltzmann distribution over responses. That is the cleanest possible statement of the reward-vs-KL trade-off from lesson 06.

Why this is a derivation, not a training algorithm — yet

$π^{*}$ is a beautiful closed form, but you cannot sample from it or train it directly, because $Z (x) = \sum_{y} π_{ref} (y) exp (r / β)$ sums over all possible responses — an intractable sum over $∣ V ∣^{T}$ sequences. This is precisely why classical RLHF resorts to PPO: it approximates $π^{*}$ iteratively without ever computing $Z$ . DPO’s genius (next section) is to make $Z$ cancel so we never need it.

Quiz 1: Where exactly in the derivation did we use that $Z (x)$ is independent of $π$ , and what breaks if it weren't?

Answer

It was used in the final step of §3.2–§3.3: after substituting $π_{ref} exp (r / β) = Z (x) π^{*}$ , we split $lo g \frac{π}{Z π ^{*}} = lo g \frac{π}{π ^{*}} - lo g Z$ and pulled $lo g Z (x)$ out of the minimization as a constant. That’s the only reason the objective reduces to “minimize $KL (π ∥ π^{*})$ plus a constant,” whose solution is trivially $π = π^{*}$ . $Z (x)$ is $π$ -independent because it is a sum over $y$ of quantities ( $π_{ref} (y ∣ x)$ , $r (x, y)$ ) that don’t involve $π$ at all — the optimization variable has been integrated out. If $Z$ depended on $π$ , you could not treat it as a constant: the " $- lo g Z (x)$ " term would contribute to the gradient w.r.t. $π$ , the problem would no longer be a plain KL-to-a-fixed-target, and $π = π^{*}$ would not be the optimum. (The same $x$ -only dependence is what makes $Z$ cancel in the next section — hold that thought.)

4. Inverting the optimum: the policy is a reward model

We have $π^{*}$ as a function of $r$ . DPO’s move is to solve for $r$ as a function of $π^{*}$ . Take logs of the boxed optimum:

$lo g π^{*} (y ∣ x) = lo g π_{ref} (y ∣ x) + \frac{1}{β} r (x, y) - lo g Z (x) .$

Isolate $r$ :

$r (x, y) = β lo g \frac{π ^{*} ( y ∣ x )}{π _{ref} ( y ∣ x )} + β lo g Z (x)$

Read this carefully. It says: any reward function $r$ is expressible in terms of its own induced optimal policy $π^{*}$ , plus a term $β lo g Z (x)$ that depends only on the prompt $x$ , not on the response $y$ . The reparameterization is exact and invertible — reward space and (optimal-policy, prompt-constant) space are in bijection. There is no information in $r$ that isn’t in $(π^{*}, π_{ref}, Z)$ .

This is the sense in which “your language model is secretly a reward model”: if you had the optimal policy for some reward, you could read that reward straight off it via $β lo g (π^{*} / π_{ref})$ , up to the prompt-only offset $β lo g Z (x)$ .

Connect to lesson 06's gauge freedom

In lesson 06 we noted the Bradley-Terry reward is identified only up to a per-prompt additive constant $c (x)$ — adding $c (x)$ to every response’s reward leaves all comparison probabilities unchanged. Here that same gauge freedom reappears as the term $β lo g Z (x)$ : it is a response-independent, prompt-dependent offset. The reparameterization respects exactly the symmetry BT already had. This is the structural reason the next step works.

5. Substitute into Bradley-Terry: the partition function cancels

Now the reward model’s whole job in lesson 06 was to feed a Bradley-Terry preference model:

$P (y_{w} ≻ y_{l} ∣ x) = σ (r (x, y_{w}) - r (x, y_{l})) .$

Bradley-Terry depends only on the difference of rewards for two responses to the same prompt $x$ . And our inverted reward has an offset that depends only on $x$ . So plug in $r (x, y) = β lo g \frac{π ^{*} ( y ∣ x )}{π _{ref} ( y ∣ x )} + β lo g Z (x)$ for both $y_{w}$ and $y_{l}$ and take the difference:

r (x, y_{w}) - r (x, y_{l}) = [β lo g \frac{π ^{*} ( y _{w} ∣ x )}{π _{ref} ( y _{w} ∣ x )} + β lo g Z (x)] - [β lo g \frac{π ^{*} ( y _{l} ∣ x )}{π _{ref} ( y _{l} ∣ x )} + β lo g Z (x)] .

The $β lo g Z (x)$ terms are identical — same prompt $x$ , same $Z (x)$ — so they subtract to zero. The intractable partition function is gone, exactly, with no approximation:

$r (x, y_{w}) - r (x, y_{l}) = β lo g \frac{π ^{*} ( y _{w} ∣ x )}{π _{ref} ( y _{w} ∣ x )} - β lo g \frac{π ^{*} ( y _{l} ∣ x )}{π _{ref} ( y _{l} ∣ x )} .$

Substitute into Bradley-Terry:

$P (y_{w} ≻ y_{l} ∣ x) = σ (β lo g \frac{π ^{*} ( y _{w} ∣ x )}{π _{ref} ( y _{w} ∣ x )} - β lo g \frac{π ^{*} ( y _{l} ∣ x )}{π _{ref} ( y _{l} ∣ x )}) .$

The preference probability is now written entirely in terms of the optimal policy — no reward model anywhere. The final step is to stop treating $π^{*}$ as an object we already have and instead parameterize it as $π_{θ}$ and fit it by maximum likelihood on the preference dataset, exactly as lesson 06 fit the reward model by MLE. Minimizing the negative log-likelihood of the observed preferences gives the DPO loss[

$L_{DPO} (θ) = - E_{(x, y_{w}, y_{l}) \sim D} [lo g σ (β lo g \frac{π _{θ} ( y _{w} ∣ x )}{π _{ref} ( y _{w} ∣ x )} - β lo g \frac{π _{θ} ( y _{l} ∣ x )}{π _{ref} ( y _{l} ∣ x )})]$

That’s the whole method. Compare it side-by-side with the reward-model loss from lesson 06 §6:

Gunjan Dhanuka — Learning Notes

Explorer

07 · DPO & RL-Free Preference Optimization

07 · DPO & RL-Free Preference Optimization

1. Learning map

2. Why this matters: the reward model was never the point

3. The centerpiece: deriving the closed-form optimal policy

3.1 The objective

3.2 Complete the square into a single KL

3.3 Read off the optimum

4. Inverting the optimum: the policy is a reward model

5. Substitute into Bradley-Terry: the partition function cancels

Graph View

Table of Contents

Backlinks