DPO: Direct Preference Optimization
RLHF works. It also requires training three models (SFT, reward model, policy), managing PPO's instability, and tuning a KL penalty. DPO asks: what if you could skip all of that? DPO directly optimizes the language model on preference pairs. No reward model. No PPO. One training loop. Same results. Implement DPO training that directly optimizes a language model on preference pairs without a separate reward model. Derive the DPO loss function and explain how it implicitly represents a reward model through the policy's log probabilities. Compare DPO vs RLHF in terms of training stability, compute cost, and number of models required. Tune the beta parameter to control how far the trained policy diverges from the reference model. You built an RLHF pipeline in Lesson 07. Three stages. Three models. The SFT model, the reward model, and the policy model optimized with PPO. The reward model alone required thousands of human preference pairs and a separate training loop. PPO required careful tuning of the KL coefficient, learning rate, clip ratio, and number of epochs. In practice, PPO training is notoriously unstable. Small hyperparameter changes cause the training to diverge. The reward model is an imperfect proxy for human preferences, and the policy finds ways to exploit its weaknesses. The KL penalty helps but requires its own tuning -- too low and…
DPO: Direct Preference Optimization: RLHF works. It also requires training three models (SFT, reward model, policy), managing PPO's instability, and tuning a…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.