RLAIF (RL from AI Feedback)

Definition

RLAIF (“RL from AI Feedback”) is a one-line edit to the RLHF pipeline: the pairwise preference labels that train the reward model are produced by an LLM judge instead of a human. Everything downstream — Bradley–Terry RM training, PPO with the KL leash — is unchanged. It turns the linear-in-dollars human-label bottleneck into a compute problem, which scales.

How it works

Given a prompt $x$ and two completions $(y_a,y_b)$, prompt a labeler LLM (“which is better? think step by step, then answer”) and read its log-probs over “A”/“B” to get a soft preference $p_{\text{AI}}(y_a\succ y_b)$. That soft label is the BT target:
$\mathcal{L}(\varphi )=-\mathbb{E}[p_{\text{AI}}\log \sigma (\Delta r)+(1-p_{\text{AI}})\log \sigma (-\Delta r)],\Delta r=r_{\varphi }(x,y_a)-r_{\varphi }(x,y_b).$

Two implementation details matter a lot (Lee et al.): chain-of-thought before the verdict improves agreement with humans; order-debias by averaging two passes with A/B swapped (LLM judges have strong position bias — not optional). Direct-RLAIF (d-RLAIF) skips the RM: prompt the labeler to score the single current sample (e.g. 1–10) directly as the PPO reward — Lee et al. report it beats canonical RLAIF, at the cost of a labeler forward-pass per RL sample.

Parity evidence (Lee et al., “RLAIF vs RLHF,” arXiv:2309.00267, ICML 2024): RLAIF is statistically indistinguishable from RLHF on summarization/helpfulness (~50% head-to-head) and better on harmlessness (88% vs 76%). Striking: RLAIF helps even when the labeler is the same size or the same checkpoint as the policy — the signal comes from judging being easier than generating, not a bigger teacher.

The judge’s biases

The labeler is a proxy with correlated, systematic errors an RL optimizer will hunt (Zheng et al., MT-Bench/Chatbot Arena, arXiv:2306.05685): position bias (favors a slot, usually first), verbosity/length bias (prefers longer answers; the “repetitive list attack”), self-preference (scores its own family higher — catastrophic if labeler and policy share a family). “AI feedback is free” is the wrong takeaway: RLAIF removes the human, not the bias — it trades a slow, diverse signal for a fast, correlated one, and correlated errors are exactly what an adversary loves.

Why it matters

RLAIF is how labeling cost is cut at frontier scale, and (with Constitutional AI) how the non-verifiable half of alignment — helpfulness, safety, style — is trained where no verifier exists. It composes with the other cost-cutters: DPO cut the optimizer, RLAIF cuts the labeler, iterated online collection keeps both calibrated. It also powers iterated online DPO (RLAIF supplies each round’s labels) and Self-Rewarding LMs (arXiv:2401.10020, the same model judging its own samples) — with the caveat that judging biases compound over rounds without an external anchor.

Taught in

• 08-scaling-rlhf-and-alternatives — RLAIF, d-RLAIF, RLAIF≈RLHF parity, the three judge biases.
• 10-frontier-and-infra — RLHF/RLAIF for the non-verifiable half of the frontier stack.

See also

• constitutional-ai — RLAIF with a constitution as the only human input
• reward-model — the proxy trained on AI labels
• bradley-terry-model — the loss fit on soft AI labels
• reward-overoptimization — what an optimizer does to judge biases
• rlhf — the pipeline RLAIF swaps the labeler in
• rl-for-llms