Capstone Lesson 40: Direct Preference Optimization from Scratch
Reward models and PPO are the classical RLHF stack. DPO collapses that stack into a single supervised loss that fits a policy directly against preference pairs. This lesson derives the DPO loss from the reward-difference identity, ships a working reference model plus policy model, computes per-token log-probabilities, and trains a tiny transformer on a preference fixture of chosen and rejected completions. Tests pin the loss math and the gradient direction so you know the implementation matches the paper. Derive the DPO loss as a sigmoid over a scaled log-ratio difference and connect it to the implicit reward. Build a reference model + policy model pair with a frozen reference and a trainable policy. Compute sequence-level log-probabilities under both models, masking prompt tokens. Train the policy on (prompt, chosen, rejected) triples and watch the chosen log-prob rise relative to rejected. Pin behaviour with tests on the loss math, the gradient sign, and the reference invariance. You have an SFT model. It follows instructions, but its outputs are uneven; some completions are clear, some are wordy or wrong. You also have a small dataset of preference pairs: for the same prompt, a human marked one completion as chosen and the other as rejected. The classical RLHF answer is a two-stage pipeline. Train a reward model on the preferences. Optimise the policy against…
Capstone Lesson 40: Direct Preference Optimization from Scratch: Reward models and PPO are the classical RLHF stack. DPO collapses that stack into a single…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.