Textbook of AI

RLAIF (Reinforcement Learning from AI Feedback) and Magpie are two synthetic-data techniques that have come to dominate post-2023 instruction-tuning and preference-learning pipelines, displacing or augmenting traditional human-labelled datasets.

RLAIF

RLAIF is a family of methods that substitute AI judges for human annotators in the preference-comparison stage of the RLHF pipeline. The seminal paper, RLAIF: Scaling RL from Human Feedback with AI Feedback (Lee, Phatale, Mansoor et al., Google Research, arXiv:2309.00267, September 2023), trained a summarisation reward model on 40,000 GPT-4-generated preferences rather than human preferences and achieved comparable downstream win-rate to RLHF on the same task.

The general RLAIF recipe:

1. Collect a prompt distribution.
2. Sample completion pairs from a base model.
3. Prompt a strong judge model (typically GPT-4 or Claude) with both completions and a rubric.
4. Use the judge's preference as the reward-model training signal.
5. RL or DPO fine-tune the base model against the resulting reward.

Subsequent RLAIF datasets include UltraFeedback (Cui et al. 2023), Skywork-Reward-Preference-80K, HelpSteer-2 (NVIDIA), Nectar (Berkeley) and PRM800K (OpenAI's process-reward dataset for math). RLAIF underpins the alignment of Zephyr, Tulu-2, Starling-LM, Notus, and most open-weight chat models released after late 2023.

Magpie

Magpie (Xu, Jiang, Niu et al., arXiv:2406.08464, June 2024) is a strikingly simple instruction-extraction method:

1. Take an aligned chat model (e.g. LLaMA-3-Instruct).
2. Feed only the system prompt and the assistant turn marker, no user input at all.
3. The model, conditioned on its instruction-tuning training, autoregressively generates a plausible user instruction to which it then continues with an assistant response.
4. Repeat at scale.

The Magpie-Pro-300K-Filtered release contains 300 K LLaMA-3-extracted instruction-response pairs at quality matching or exceeding manually curated SFT datasets. Magpie-Air and Magpie-Reasoning variants extend the technique to other base models and to chain-of-thought reasoning data.

Licensing

RLAIF datasets follow the licence of their judge model's outputs: UltraFeedback is MIT-licensed but contains GPT-4 derivatives; Magpie outputs are MIT-licensed but contain LLaMA-3-Instruct derivatives and inherit the LLaMA-3 community licence. The legal status of training competitor models on these synthetic-derivative datasets remains the same unsettled question that haunts ShareGPT and UltraChat.

Models trained on RLAIF / Magpie

Magpie-Llama-3-8B-Pro, Llama-3-Magpie-8B, Phi-3-Magpie, Qwen-2-Magpie, Hermes-3, Tulu-3, and a long tail of academic instruction-tuned models. RLAIF data is now standard in essentially all open chat-model recipes.

Significance

Magpie and RLAIF together completed the closure of the synthetic-data loop: instruction prompts, assistant completions, and preference judgments can all now be produced without human input, given API or open-weight access to a strong frontier model. The result is order-of-magnitude cost reduction in alignment training and a corresponding shift in the industry's data bottleneck from human annotation to frontier-model API access.