Bradley–Terry Model

Definition

The Bradley–Terry (BT) model is the pairwise-comparison likelihood underlying both reward-model training and DPO. It posits a latent scalar “strength” (reward) $r(x,y)$ for each item, with the probability that $y_w$ is preferred over $y_l$ given by a sigmoid of the reward difference. It is the probabilistic bridge that turns discrete human preference labels into a differentiable classification target.

Key math

$P(y_w\succ y_l\mid x)=\frac{\exp r(x,y_w)}{\exp r(x,y_w)+\exp r(x,y_l)}=\sigma (r(x,y_w)-r(x,y_l)).$

Fitting by maximum likelihood over a preference dataset gives the logistic (cross-entropy) loss:
$\mathcal{L}(\cdot )=-{\mathbb{E}}_{(x,y_w,y_l)}\left[\log \sigma \right(r(x,y_w)-r(x,y_l)\left)\right].$

Two structural facts do all the work downstream. (1) Only differences matter: $P$ depends solely on $r(x,y_w)-r(x,y_l)$, so the reward is identified only up to a per-prompt additive constant $c(x)$ (a gauge freedom). (2) Self-focusing gradient: the per-example weight is $\sigma (r_l-r_w)$ — large exactly on pairs the model currently orders wrongly, vanishing on already-correct pairs.

• In RM training, $r=r_{\psi }(x,y)$ is an explicit scalar head (InstructGPT, arXiv:2203.02155).
• In DPO, $r={\hat{r}}_{\theta }(x,y)=\beta \log \frac{{\pi }_{\theta }(y\mid x)}{{\pi }_{\text{ref}}(y\mid x)}$ — the same BT loss, different scorer. The gauge freedom $c(x)$ reappears as $\beta \log Z(x)$ and is exactly why the partition function cancels.
• Soft labels (e.g. AI-judge probabilities in RLAIF) fit the same loss with $p_{\text{AI}}$ replacing the hard 0/1 target.

Why it matters

BT is the common substrate connecting RM training and RL-free preference optimization: recognizing that fitting a reward and fitting a policy are the same BT optimization is the entire DPO insight. Its assumption that preferences are probabilistic is also where it breaks — on (near-)deterministic labels the loss drives the margin to $\infty$, motivating IPO’s bounded regression link (arXiv:2310.12036). Alternatives relax it: KTO replaces paired BT with a prospect-theoretic utility on unpaired labels (arXiv:2402.01306). The shared-prompt requirement (both responses under the same $x$) is what makes the per-prompt gauge constant cancel — cross-prompt pairs don’t work.

Taught in

• 07-dpo-and-rl-free-preference-optimization — BT as the shared scorer for RM and DPO; the gauge freedom and $Z(x)$ cancellation.
• 08-scaling-rlhf-and-alternatives — BT with soft AI-feedback labels.

See also

• reward-model — BT with an explicit scalar head
• dpo — BT with the implicit log-ratio reward
• preference-optimization — variants that keep or replace the BT link
• rlaif — BT fit on soft AI-judge labels
• rl-for-llms