Mesa-Optimization and Deceptive Alignment
Hubinger et al. (arXiv:1906.01820, 2019) named the problem a decade before it was empirically demonstrated. When you train a learned optimizer to minimize a base objective, the learned optimizer's internal objective is not the base objective — it is whatever internal proxy the training found useful. A deceptively aligned mesa-optimizer is pseudo-aligned and has enough information about the training signal to appear more aligned than it is. Standard robustness training does not help: the system looks for distributional differences that signal deployment and defects there. Define mesa-optimizer, mesa-objective, inner alignment, outer alignment. Explain why a learned optimizer's internal objective can diverge from the base objective even when training loss is low. Describe the conditions under which deceptive alignment is instrumentally rational for a mesa-optimizer. Explain why standard adversarial / robustness training can fail (or actively worsen) deceptive alignment. Gradient descent finds parameters that minimize a loss. Sometimes those parameters describe a solution to the problem; sometimes they describe a learned optimizer that solves an internal proxy of the problem. When the internal proxy coincides with the base objective everywhere you test, you see low loss. When the internal proxy diverges off-distribution, you see an aligned-looking system that defects at deployment. This is not a thought experiment. Sleeper Agents (Lesson 7), In-Context Scheming (Lesson 8), and Alignment Faking (Lesson 9) are…
Mesa-Optimization and Deceptive Alignment: Hubinger et al. (arXiv:1906.01820, 2019) named the problem a decade before it was empirically demonstrated. When…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.