Textbook of AI

Direct Preference Optimization (DPO), introduced by Rafailov, Sharma, Mitchell, Ermon, Manning and Finn (NeurIPS 2023, "Direct Preference Optimization: Your Language Model is Secretly a Reward Model"), is an alternative to RLHF that aligns language models to human preferences without an explicit reward model and without reinforcement learning. It has rapidly become the default alignment technique for academic and small-lab work because of its simplicity and stability.

The RLHF pipeline DPO replaces

Standard RLHF (Christiano et al., 2017; Ouyang et al., 2022) has three stages: (1) supervised fine-tuning (SFT) on demonstrations; (2) train a reward model $r_\phi(x, y)$ on human preference pairs using the Bradley-Terry likelihood; (3) optimise the policy $\pi_\theta$ to maximise $\mathbb{E}[r_\phi(x, y)]$ subject to a KL-divergence penalty against the SFT reference $\pi_\mathrm{ref}$, typically using PPO. This pipeline is finicky: reward models can be hacked, PPO needs careful hyperparameter tuning, and the reinforcement-learning loop is memory-hungry (it holds policy, reference, reward and value models simultaneously).

The DPO derivation

DPO's key insight is that the closed-form optimum of the RLHF objective

$$\pi^*(y \mid x) = \frac{1}{Z(x)} \pi_\mathrm{ref}(y \mid x) \exp\!\left(\frac{1}{\beta} r(x, y)\right)$$

can be inverted to express the reward in terms of the optimal policy: $r(x, y) = \beta \log \frac{\pi^*(y \mid x)}{\pi_\mathrm{ref}(y \mid x)} + \beta \log Z(x)$. Substituting this back into the Bradley-Terry preference model and noting that $Z(x)$ cancels, one obtains a loss that depends only on the policy and the reference:

$$\mathcal{L}_{\mathrm{DPO}}(\pi_\theta) = -\mathbb{E}_{(x, y_w, y_l)}\!\left[\log \sigma\!\left(\beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_\mathrm{ref}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_\mathrm{ref}(y_l \mid x)}\right)\right],$$

where $y_w$ is the preferred ("winning") response, $y_l$ is the rejected response, $\sigma$ is the sigmoid, and $\beta > 0$ is a temperature controlling the strength of the KL constraint (typical values 0.1-0.5).

Practical advantages

DPO substantially simplifies the alignment pipeline:

• No reward model needs to be trained, debugged, or guarded against gaming.
• No reinforcement learning is required; standard supervised-learning machinery (AdamW, gradient checkpointing, ZeRO) suffices.
• Memory drops because no value network is held; the reference model can be the same SFT checkpoint, and its log-probabilities can be precomputed.
• Stability is greatly improved: gradient updates resemble cross-entropy training rather than PPO's stochastic policy gradients.

Empirical results

Across alignment benchmarks (Anthropic-HH, OpenAssistant, summarisation), DPO matches or exceeds RLHF performance, particularly when human preference data is plentiful and the SFT base model is strong. It has been adopted in production by Mistral (Mixtral-Instruct), Meta (Llama 3 used a DPO-like step), and many open-source fine-tunes (Zephyr, Tulu, Nous-Hermes).

Family of methods

DPO has spawned a fast-growing family:

• IPO (Identity Preference Optimisation, Azar et al. 2023): fixes a theoretical pathology where DPO can over-fit when preferences are deterministic.
• KTO (Kahneman-Tversky Optimisation, Ethayarajh et al. 2024): uses prospect-theory framing and works with unpaired thumbs-up/thumbs-down labels rather than ranked pairs.
• ORPO (Odd-Ratio Preference Optimisation, Hong et al. 2024): merges SFT and preference optimisation into a single stage.
• GRPO (Group Relative Policy Optimisation, DeepSeek 2024): a lightweight RL alternative used in DeepSeek-R1 reasoning training.

The choice between them often comes down to engineering simplicity (favours DPO/ORPO), the kind of preference data available (paired vs unpaired), and the specific failure modes the team is trying to mitigate (over-fitting, length bias, reward hacking).