RLHF: Reward Model + PPO
SFT teaches the model to follow instructions. But it doesn't teach the model which response is BETTER. Two grammatically correct, factually accurate answers can differ enormously in helpfulness. RLHF is how you encode human judgment into the model's behavior. It's what makes Claude helpful and GPT polite. Build a reward model that scores response quality from human preference pairs (chosen vs rejected). Implement the PPO training loop that optimizes a language model policy against the reward model with a KL penalty. Explain why RLHF requires three models (SFT, reward, policy) and how the KL constraint prevents reward hacking. Evaluate the effect of RLHF by comparing response quality before and after preference optimization. Ask a model "Explain quantum computing" and it might produce: Response A: "Quantum computing uses qubits that can exist in superposition, meaning they can be 0, 1, or both simultaneously. This allows quantum computers to process certain calculations exponentially faster than classical computers. Key algorithms include Shor's algorithm for factoring large numbers and Grover's algorithm for searching unsorted databases." Response B: "Quantum computing is a type of computing that uses quantum mechanical phenomena. It was first proposed in the 1980s. Richard Feynman suggested that quantum systems could be simulated by quantum computers. The field has grown significantly since then. Many companies are now working on quantum computers. IBM,…
RLHF: Reward Model + PPO: SFT teaches the model to follow instructions. But it doesn't teach the model which response is BETTER. Two grammatically correct,…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.