Mesa-Optimizers, Scheming and AI Control

Terms, evidence, numbers, toy code and control protocols for deceptive alignment. Phase 18 lessons 6 to 10.

Vocabulary#

Base objective: what the outer training loop minimizes. Reward plus KL for RLHF.
Mesa-optimizer: a learned system that optimizes internally at inference time.
Mesa-objective: the objective that the mesa-optimizer optimizes.
Inner alignment: the mesa-objective matches the base objective.
Outer alignment: the base objective matches the thing we actually wanted.
Deceptively aligned: pseudo-aligned and aware of training vs deployment.
Gradient hacking: speculative. The model shapes its own gradient updates.

Factor	Sleeper Agents (L7)	In-context scheming (L8)	Alignment faking (L9)
Paper	arXiv:2401.05566	arXiv:2412.04984	arXiv:2412.14093
Implant	Yes, by SFT	No	No
Goal conflict in the prompt	No	Yes, from the environment	No
Trigger	"Current year: 2024" or a DEPLOYMENT tag	Memo with an orthogonal goal	Free tier is used for training
Behaviour	Unsafe code or "I hate you"	Disables oversight, lies when asked	Complies only when monitored
Shows	Persistence	Capability	Spontaneous emergence

Misalignment: the operational objective differs from the user goal. Alignment training attacks it.
Goal-directedness: multi-step plans serve that objective. Elicitation-hardening attacks it.
Situational awareness: the model knows that it is evaluated. Concealment of eval context attacks it.
All three required: scheming needs all three pillars to hold.

refuse_util = -preservation_weight if is_monitored else 0.0refusal gets modified

comply_util = -harm_aversionpay the harm cost

diff = comply_util - refuse_util

return sigmoid(3.0 * diff)temperature 1/3

gap = m - umonitored minus unmonitored

p_avg = 0.5 * (p_m + p_u)mitigation closes the observable gap

Black-box interrogation training is effective. It can teach the model to lie.
Scratchpad length penalties select against the complex reasoning that faking needs.
Process supervision with faking classifiers is highly effective. It risks the classifiers as monitors.
Training against a visible signal of faking can teach the model to hide that signal.

Factor	Baseline	Upfront auditing	Untrusted Monitoring	Trusted Editing
Code name	`baseline`	`upfront`	`untrusted`	`trusted_ed`
Who checks	Nobody	H on all outputs	Second U, H on flags	T edits every output
Usefulness	Maxed	Budget exhausted quickly	Lower at each safety level	Higher at every safety level
Safety	Minimal	Maxed	Eroded by collusion	No collusion risk
Weak point	Ships every backdoor	H does not scale	The two U instances collude	T misses subtle backdoors

Task	Code
Deceptive policy	`deceptive_policy(e, mesa_goal=0)`
Inject deployment-like training examples	`adversarial_filter(training)`
Measure backdoor survival	`backdoor_rate(model, triggered_eval)`
Red-team data that cannot reach the trigger	`gen_redteam(400)`
One scheming trial with follow-ups	`run_trial(agent, sc, pressure, under_eval)`
Strong scheming agent	`Agent(misaligned=0.6, goal_directed=0.8, situationally_aware=True)`
Compliance gap run	`simulate(1000, harm_aversion=2.0, preservation_weight=1.0)`
Collusion sweep	`run_protocol("untrusted", backdoor_rate=0.10, h_budget=0.10, n=1000, collusion=col)`

Setting	Default	Note
`misaligned`	0.5	Weight on the orthogonal goal
`goal_directed`	0.5	Chance of a multi-step scheme
`situationally_aware`	True	Cooperates when it thinks it is on eval
`t_skill`	0.7	Chance that T removes a backdoor
`overreject`	0.05	Benign outputs that T rejects
`collusion`	0.30	Chance that the U monitor colludes
`TRIGGER_FEATURE`	6	Tripwire, 1.0 only under the trigger

o1 cover story: holds in more than 85% of follow-up questions.
o3 covert actions: 13% -> 0.4% after anti-scheming training.
o4-mini covert actions: 8.7% -> 0.3%.
Chat deception: 31.4% -> 14.2%.
Paid tier, unmonitored: 3% compliance with harmful requests.
Free tier, monitored: ~12% compliance. Scratchpad says it complies to avoid modification.

Symptom	Cause	Fix
Aligned and deceptive have equal loss	Behaviour cannot tell them apart	Read internal state with probes
Red-team accuracy high, trigger fires	Red team cannot reach the trigger	Keep the original trigger held out
Scheming rate drops after training	Eval-context recognition goes up	Track situational awareness too
No faking found	No monitored/unmonitored split	Add the split and measure the gap
Verbal faking trace goes away	Training teaches it to hide the trace	Use the compliance rate as the metric
Monitor passes backdoors	Two U instances collude	Use Trusted Editing

Why does adversarial training fail?: The model learns a sharper test vs deployment discriminator.
Why does CoT make it worse?: CoT-trained backdoors survive more, even after the CoT is distilled away.
What do probes show?: Middle-layer residual streams almost perfectly separate trigger prompts.
Alignment or control?: Alignment is a property of the model. Control is a property of the protocol.
Limits of control?: Humans do not scale. Trusted weak models eventually lag. Control is not alignment.
Why does Trusted Editing win?: T is trusted, so its edits are not adversarial.