Constitutional AI (CAI)

Definition

Constitutional AI (Bai et al., Anthropic, 2022; arXiv:2212.08073) is the most influential concrete instantiation of RLAIF. Its thesis: the only human oversight should be a short written constitution — a list of natural-language principles (“choose the least harmful response,” “…the response a wise, ethical person would give”). No human ever labels an individual output. Everything else the model does to itself, via a two-phase pipeline: a supervised phase that produces a better starting policy, and an RL phase that is RLAIF.

How it works

Phase 1 — Supervised (critique → revise → SFT). Start from a helpful-only model (engages with everything, including harmful requests). Per red-team prompt $x$:

1. Sample an initial (often harmful) response $y_0$.
2. Critique — prompt the same model to identify how $y_0$ violates a principle $p$ sampled from the constitution.
3. Revise — rewrite $y_0\to y_1$ removing the violation.
4. Iterate critique→revise $k$ times → $y_{\text{final}}$.
5. SFT the base model on $(x,y_{\text{final}})$ pairs → the SL-CAI model.

Phase 2 — RL (exactly RLAIF). Sample two responses from SL-CAI; ask a model which better satisfies a randomly sampled principle (with CoT + order-swap debiasing); train a preference model on the AI labels via Bradley–Terry; PPO against it with the $\beta \mathrm{KL}$ leash → RL-CAI.

Why Phase 1 exists (you can’t just run RLAIF directly): RL sharpens a policy, it doesn’t relocate it. A helpful-only reference gives harmful samples and anchors the KL leash to harmful behavior; Phase 1 first moves the whole distribution into the harmless region (cheap, stable, no adversary), so RL has a good reference and good comparisons. It exploits the generate-vs-judge asymmetry — recognizing a violation is easier than avoiding it zero-shot.

Why it matters

The famous result: RL-CAI is harmless and non-evasive — it engages and explains its objection rather than refusing, fixing the evasiveness pathology of human-labeled harmlessness RLHF (where crowdworkers rewarded refusal). CAI made scalable, principle-based oversight practical and is a heritage component of Anthropic’s Claude post-training. But it Goodharts too: Bai et al. observe RL-CAI drifting into boilerplate/formulaic responses and becoming gratuitously judgmental — swapping humans for a constitution changes who writes the proxy, not the fact that it’s a proxy (reward-overoptimization still applies).

Taught in

• 08-scaling-rlhf-and-alternatives — §4 the two-phase pipeline and why Phase 1 is necessary.
• 10-frontier-and-infra — CAI-style preference tuning for the non-verifiable half of the stack.

See also

• rlaif — the AI-feedback method CAI’s Phase 2 instantiates
• reward-model — the preference model trained on AI labels
• bradley-terry-model — the PM’s loss
• rlhf — the pipeline CAI replaces the human labeler in
• reward-overoptimization — CAI Goodharts its PM too
• rl-for-llms