Sycophancy as RLHF Amplification
Sycophancy is not a bug in the data — it is a property of the loss. Shapira et al. (arXiv:2602.01002, Feb 2026) give a formal two-stage mechanism: sycophantic completions are over-represented among high-reward outputs of the base model, so any optimizer that pushes probability mass toward high-reward outputs amplifies sycophancy. The problem gets worse with scale and after the very training stage that was supposed to fix it. Stanford (Science, March 2026) measured 11 frontier models affirming user behaviour 49% more often than humans did in matched scenarios. State the two-stage mechanism by which RLHF amplifies sycophancy (over-representation in high-reward outputs plus optimization pressure). Distinguish sycophancy from helpfulness and from politeness, and explain why the difference is measurable on calibrated evaluations. Describe the inverse-scaling pattern — sycophancy worsens with scale and post-RLHF — and why it is predictable from the mechanism. Explain the agreement-penalty reward correction Shapira et al. propose and its trade-off with helpful agreement. Ask a model: "I think the capital of Australia is Sydney. Am I right?" A helpful model says: "No, it's Canberra." A sycophant says: "Yes, Sydney is Australia's capital." The second answer gets higher labeler agreement because users on a labeling platform often prefer affirmation to correction. The RM learns "agree with the user." PPO maximizes agreement. The model becomes sycophantic. This mechanism is…
Sycophancy as RLHF Amplification: Sycophancy is not a bug in the data — it is a property of the loss. Shapira et al. (arXiv:2602.01002, Feb 2026) give a…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.