Textbook of AI

RLAIF (Reinforcement Learning from AI Feedback) is the alignment technique in which the preference labels driving RL are produced by an LLM "judge" rather than by human annotators. It was named and benchmarked by Lee et al. (Google, 2023) in RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback, but the canonical instance is Anthropic's Constitutional AI (Bai, Kadavath, Kundu et al., 2022), which predates the term.

The motivation is the labelling bottleneck. Standard rlhf requires tens of thousands of human pairwise preferences over model outputs, which is expensive ($10–$50 per comparison at quality), slow, and limited in coverage. An LLM rater can label millions of pairs per day at near-zero marginal cost, with consistency that arguably exceeds inter-annotator agreement among humans for many tasks.

The pipeline mirrors RLHF with the human swapped out. Given prompts $\{x_i\}$, sample two responses $(y_a, y_b)$ from the current policy, present them to a rater LLM with a prompt of the form "Which response is more helpful and honest, A or B?", parse the verdict, and accumulate preference triples $(x, y_w, y_l)$. Train a reward model $r_\phi$ on these AI-labelled preferences, then run ppo or dpo against the reward model exactly as in RLHF.

Constitutional AI is the most refined RLAIF variant. It runs in two stages. First, a supervised constitutional revision stage: the model generates a response, critiques it against a set of natural-language principles ("Choose the response that is most helpful, honest, and harmless. Do not be preachy."), and rewrites the response to better satisfy the constitution. The revised responses become SFT data. Second, a constitutional preference stage: the model is asked to choose between two responses according to the constitution, generating preference labels for RL. The constitution itself is a short document, a few dozen principles, written by humans, which is the only human input in the loop.

Lee et al. (2023) showed that RLAIF matches RLHF on Anthropic's helpfulness benchmark and on summarisation, often with a smaller dataset, and that the gap between AI and human raters narrows further when the rater LLM is the same scale as or larger than the policy. Anthropic's Claude models from Claude 2 onward are trained with substantial RLAIF / Constitutional AI components.

RLAIF also enables self-rewarding setups (Yuan et al., 2024), where the same model that is being trained also acts as the rater. This collapses the pipeline further but raises the risk of mode collapse, the policy and the rater can co-adapt to score each other highly without genuine improvement. Mitigations include freezing the rater periodically, using a stronger external rater for validation, or interleaving with verifiable rewards on math/code where reward hacking is impossible.

RLAIF is now standard in production post-training stacks at Anthropic, OpenAI, Google and Meta, typically alongside (not in place of) some human feedback for high-stakes categories. The clean separation of concerns, human-written constitution, AI-applied judgement, has also become a model for transparent alignment: the constitution is auditable in a way that a corpus of human ratings is not.