Constitutional AI and RLAIF
Bai et al. (arXiv:2212.08073, 2022) asked: what if we replaced the human labeler with an AI that reads a list of principles? Constitutional AI has two phases — self-critique and revision under a constitution, then RL from AI Feedback. The technique coined the term RLAIF and shipped in the Claude 1 post-training pipeline. On 21 January 2026 Anthropic published a rewritten Claude constitution: explanatory reasoning over prescriptive rules, a four-tier priority hierarchy, and the first major-lab formal acknowledgment of uncertainty about model moral status. Released under CC0 1.0. Describe the two phases of Constitutional AI (critique-and-revise SFT, RL from AI feedback) and the role of the constitution in each. Explain why replacing a human preference labeler with an AI labeler is not a "cheaper" RLHF — it changes which failure modes the pipeline has. Summarize the four-tier priority structure of the 2026 Claude constitution and what changed from the 2023 rewrite. Describe Constitutional Classifiers and the drop from 23.7% compute overhead (v1) to 1% (v2 / 2026). RLHF needs labelers. Labelers are slow, biased, and expensive. You can eliminate a labeler by replacing them with a model that reads explicit principles. The first formal version of this substitution was Bai et al.'s Constitutional AI. It worked well enough that every frontier lab now uses some variant of AI-feedback post-training. The…
Constitutional AI and RLAIF: Bai et al. (arXiv:2212.08073, 2022) asked: what if we replaced the human labeler with an AI that reads a list of principles?…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.