Constitutional AI and Self-Improvement
RLHF needs humans in the loop. Constitutional AI replaces most of them with the model itself. Write a list of principles, have the model critique its own outputs against those principles, and train on the critiques. DeepSeek-R1 pushed this further in 2025: let the model generate millions of reasoning traces, grade them with a rule, and run GRPO on the outcome. Most of the "alignment work" in a 2026 frontier model is the model alignment itself. This lesson builds both loops. Implement the Constitutional AI two-stage loop: self-critique plus self-revision, then preference training on the revised pairs. Derive the GRPO objective (DeepSeek-R1's group-relative policy optimization) and contrast it with PPO's value-function baseline. Generate verifiable reasoning traces with rule-based outcome rewards and score them without a separate reward model. Decide when self-improvement beats human preference data and when it collapses into mode seeking. You built RLHF in Lesson 07 and DPO in Lesson 08. Both depend on the same expensive input: human preference pairs. Anthropic's InstructGPT-era pipeline used roughly 33,000 comparisons. Llama 2 Chat used over 1.5 million. Claude 3 used more. This data is slow, expensive, and biased toward whatever the annotators happened to believe on the day they were rating. The 2022 Constitutional AI paper asked a simple question. What if the model generates the preference labels itself? Give…
Constitutional AI and Self-Improvement: RLHF needs humans in the loop. Constitutional AI replaces most of them with the model itself. Write a list of…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.