Instruction-Following as Alignment Signal
Every later critique of RLHF argues against this pipeline. Before you study how optimization pressure distorts a proxy, you have to see the proxy. InstructGPT (Ouyang et al., 2022) defined the reference architecture: supervised fine-tuning on instruction-response pairs, a reward model trained on pairwise preference rankings, and PPO against the reward model with a KL penalty to the SFT policy. A 1.3B InstructGPT was preferred over a 175B GPT-3. That single result is the reason every frontier lab in 2026 still ships an RLHF-shaped post-training pipeline. Name the three stages of the InstructGPT pipeline and the loss used in each. Explain why a 1.3B instruction-tuned model beat the raw 175B GPT-3 on human preference evaluation. State what the KL penalty in stage 3 is protecting against and why removing it collapses to mode-seeking behaviour. Describe the alignment tax and the PPO-ptx mitigation Ouyang et al. used against it. Pre-trained language models complete text. They do not answer questions. Ask GPT-3 "write a Python function that reverses a list" and you often get back another prompt, because most of the training distribution is web text that continues with more web text. The model is doing its job — the job is wrong. The proxy every serious lab used to fix this is human preference. Two completions go to a rater; the…
Instruction-Following as Alignment Signal: Every later critique of RLHF argues against this pipeline. Before you study how optimization pressure distorts a…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.