Preference Optimization (RL-free family)

Definition

Preference optimization is the umbrella for the RL-free family of alignment methods that fit a policy directly on preference data — no reward model, no PPO loop — of which DPO is the prototype. Every member replaces “find the policy that maximizes a reward” (RL) with “express the reward via the policy and fit by supervised loss.” The variants (IPO, KTO, ORPO, SimPO, CPO) are not seven unrelated algorithms but points in a small design space, each a targeted patch on one crack in vanilla DPO.

The design axes

RainbowPO (ICLR 2025) organizes the family along a few axes:

• (i) Link function — DPO/most use $-\log \sigma$ (Bradley–Terry); IPO uses a bounded squared regression loss to a target margin.
• (ii) Reference vs reference-free — DPO/IPO/KTO keep ${\pi }_{\text{ref}}$ (inside the objective, not just a leash); ORPO/SimPO/CPO drop it (halve memory, remove reference-sensitivity).
• (iii) Length normalization — SimPO uses the length-normalized average log-prob (attacks DPO’s verbosity/length bias).
• (iv) SFT/NLL anchor — ORPO/CPO add a behavior-cloning term on $y_w$ (pins chosen-likelihood, counters likelihood displacement).
• (v) Paired vs unpaired — KTO drops the shared-prompt pair requirement (works on thumbs-up/down).

Method	Ref-free	Paired	Core change	Fixes	arXiv
DPO	no	yes	BT on $\beta \log ({\pi }_{\theta }/{\pi }_{\text{ref}})$	baseline	2305.18290
IPO	no	yes	squared/regression loss to margin $\frac{1}{2\beta }$	BT overfit on (near-)deterministic prefs	2310.12036
KTO	no*	no	prospect-theory utility on unpaired labels	removes paired-data need; loss-aversion	2402.01306
ORPO	yes	yes	odds-ratio term added to SFT loss	one-stage, no ref; resists displacement	2403.07691
SimPO	yes	yes	length-normalized avg log-prob + margin	drops ref + DPO length bias	2405.14734
CPO	yes	yes	contrastive + BC-NLL anchor on $y_w$	ref-free; NLL pins chosen-likelihood	2401.08417

*KTO still uses a reference for its implicit reward but needs no paired data.

Why it matters

Preference optimization is the default for cheap, accessible, stable alignment — it deletes RLHF’s operational machinery while reusing its preference data. Pick the variant by constraint, not leaderboard: unpaired feedback → KTO; single-stage / memory-bound → ORPO; length bias → SimPO; deterministic/synthetic prefs overfitting → IPO; want a safety hedge on chosen-likelihood → CPO/ORPO’s NLL anchor. RainbowPO cautions that “beats DPO” claims are setup-sensitive (DPO’s $-\log \sigma$ link often empirically beats IPO’s regression despite weaker theory). All inherit DPO’s offline distribution-shift limitation, cured by going iterative/online.

Taught in

• 07-dpo-and-rl-free-preference-optimization — §9 the variant zoo and the unifying design-space view.

See also

• dpo — the prototype and derivation
• bradley-terry-model — the link most variants build on
• reward-model — the explicit proxy this family avoids
• kl-regularization-rlhf — the leash DPO/IPO/KTO keep, ref-free variants drop
• rlhf — the pipeline being simplified
• rl-for-llms