The Direct Preference Optimization Family
Rafailov et al. (2023) showed RLHF's optimum has a closed form in terms of the preference data, so you can skip the explicit reward model and optimize the policy directly. That insight spawned a family — IPO, KTO, SimPO, ORPO, BPO — each fixing a failure mode of DPO. In 2026, direct alignment algorithms ship more frontier post-training runs than PPO. But the over-optimization curve from Lesson 2 still applies: DAAs do not escape Goodhart, they just move where it bites. Derive the DPO closed form from the RLHF-with-KL optimum. State the failure mode each of IPO, KTO, SimPO, ORPO, BPO fixes in DPO. Distinguish "implicit reward gap" from "preference strength" and explain why IPO's identity mapping matters. Explain why Rafailov et al. (NeurIPS 2024) prove DAAs over-optimize despite having no explicit RM. The RLHF objective (Lesson 1): has a known optimum: So the reward is implicitly defined by the ratio of the optimal policy to the reference: Substitute this into the Bradley-Terry preference likelihood and the partition function Z(x) cancels because it depends only on x. What remains is a loss in the policy parameters alone — no reward model needed. That is DPO. The wrinkle: the derivation assumes the optimum is reachable, the preference data is in-distribution, and the reference policy is the true mode anchor. None of…
The Direct Preference Optimization Family: Rafailov et al. (2023) showed RLHF's optimum has a closed form in terms of the preference data, so you can skip the…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.