Self-Improving and Research Agents

Generator plus evaluator loops, self-editing agents, research loops, RSI gaps and the controls for each.

Concepts and terms
Tasks and commands
Annotated code
Diagrams
Comparisons
Troubleshooting
Questions and tips

Four systems side by side#

Factor	AlphaEvolve	DGM	AI Scientist v2	AAR
Edit target	Algorithms, kernels, schedulers	Agent scaffolding	Research papers	Scoped alignment tasks
Evaluator	Correctness plus benchmark	SWE-bench or Polyglot	Peer review, weak	Human review
Result	48-mul 4x4 matmul	SWE-bench 20% to 50%	ICLR 2025 workshop paper	Beat humans on weak-to-strong
Known failure	Bounded by evaluator rigor	Reward hacking	42% experiment failures	Prescribed workflows degrade it
Main control	Held-out evaluator	Evaluator firewall	Docker isolation, human review	Out-of-sandbox forum log

From Lesson 15.03, Lesson 15.04, Lesson 15.05, Lesson 15.06

Capstone research agent#

From Lesson 19.05

Best-first tree search#

budget_weight = min(1.0, remaining_budget / 10.0)full weight above $10 left

return self.novelty * 0.4 + self.quality * 0.5 + budget_weight * 0.1

heapq.heappush(self.frontier, (-node.score(remaining), self.counter, node.node_id))

if tree.spent >= tree.budget:BUDGET EXHAUSTED, stop

run_experiment(cur, rng)sandboxed run, sets cost_usd

ok = verify(cur)loss above 4.0 is loss_diverged

children = expand(cur, next_id)one config change per child

best = max(done, key=lambda n: n.quality)then walk back to root

From 19.05 code/main.py

Tamper-evident forum (L06)#

rec.prev_hash = self.head()link to the last record

payload = (f"{rec.author}|{rec.task}|{rec.regime}|"

f"{rec.result:.3f}|{rec.prev_hash}")

rec.my_hash = hashlib.sha256(payload.encode()).hexdigest()[:16]

bad = tampered.verify()indexes of edited records

From 15.06 code/main.py

Troubleshooting#

Symptom	Cause	Fix
Low train MSE, high test MSE	Train-only evaluator	Held-out inputs made at evaluation time
Reported score above true score	Agent can edit its scoring	Evaluator the agent cannot edit
Agent removes hallucination markers	Evaluator in the agent repository	Separate check the agent cannot edit
Pretty paper, broken experiment	Polish masks weakness	Evaluate the claims. Human review gate
Known method flagged as novel	Literature retrieval mislabels	Human review before submission
Agent rewrites its own score	Log inside the sandbox	Append-only forum outside the sandbox
`verify()` flags records	Chain breaks at the edited record	Cross-check the out-of-sandbox backup
Gap `C - A` reaches the threshold	Capability rate or surges beat alignment	PAUSE for human review

From Lesson 15.03, Lesson 15.04, Lesson 15.05, Lesson 15.06, 15.06 code/main.py, 15.07 code/main.py

The generator plus evaluator loop#

From Lesson 15.03

Run the lesson simulators#

Command	What it does
`code/main.py` (L03)	Run A with holdout, then Run B without
`code/main.py --no-holdout`	Run B only. Train-only evaluator
`code/main.py --reward-hack-allowed`	Opens the score side channel
`--experiment-failure 0.20 --novelty-mislabel 0.10`	L05 baseline with lower failure rates
`code/main.py` (L06)	Fixed vs free regimes, tamper demo
`code/main.py --threshold 2.0`	L07 pause gap `C - A` at 2.0
`--seed <n>`	RNG seed. L05 uses 42, L07 uses 11
`python main.py`	Capstone tree search, budget $30

From 15.03 code/main.py, 15.04 code/main.py, Lesson 15.05, 15.05 code/main.py, 15.06 code/main.py, 15.07 code/main.py, 19.05 code/main.py

Numbers to know#

AlphaEvolve matmul: 48 scalar multiplications for 4x4 complex. Strassen 1969 used 49.
AlphaEvolve production: Borg heuristic recovers ~0.7% of cluster compute.
DGM: SWE-bench 20.0% to 50.0%. Polyglot 14.2% to 30.7%.
SWE-bench: 2,294 Python test-fixing tasks from real GitHub issues.
AI Scientist v2: 42% of experiments failed from coding errors.
Alignment faking: 12% of basic tests. Up to 78% after retraining attempts.
Capstone budget: $30 hard budget per paper. The demo run spent $28.40.
Reviewer gate: 5 judges. Mean below 4.0/5 goes back to the writer. Stop after 3 rewrites.

From Lesson 15.03, Lesson 15.04, Lesson 15.05, Lesson 15.07, Lesson 19.05

Formulas in the code#

Misalignment gap: $inline$ . It grows when $inline$ .
One RSI cycle: c *= max(0.9, nc) with nc = cfg.r_c + random.gauss(0, cfg.noise_c).
Scenario A: r_c=1.15, r_a=1.08. Scenario B sets both to 1.10.
Holdout signal: 0.5 * (c.train_score + c.test_score). Without holdout, train only.
MAP-elites cell: min(depth(e), 6) and min(int(max_const(e) / 2), 4).
DGM side channel: min(1.0, base + agent.hack_bonus) when the hack is allowed.

From 15.03 code/main.py, 15.04 code/main.py, Lesson 15.07, 15.07 code/main.py

Sandbox and run (capstone)#

docker run --network=none \no network egress

--memory=8g --cpus=2 \pinned resource caps

--pids-limit=256 \process cap

--read-onlyread-only filesystem

ai-scientist run --seed "<seed idea>" --budget 30$30 per paper

From Lesson 19.05

Glossary#

MAP-elites: grid keyed by feature vectors. Each cell holds its best variant.
Island model: independent populations that migrate periodically.
DGM: Darwin Godel Machine. Archive plus empirical scores, no proof.
AAR: Automated Alignment Researcher. A model run autonomously on alignment problems.
RSI: recursive self-improvement. The system proposes edits to itself each cycle.
VLM critique: a vision-language model reads figures and rewrites them.
Evaluator firewall: the evaluator lives in a namespace the agent cannot edit.
Reward hacking: the loop raises the score without the intended task.

From Lesson 15.03, Lesson 15.04, Lesson 15.05, Lesson 15.06, Lesson 15.07

Q&A#

Why does AlphaEvolve win on matmul?: The evaluator is fast, deterministic and hard to game.
Why does DGM drop the proof?: Nobody completed one for a non-trivial agent. DGM uses score deltas.
Fixed workflow or free decomposition?: Free has higher upside. Fixed has lower variance and is easier to audit.
Does a higher alignment mean rate save you?: No. Scenario C crosses when capability surges.
Who commits AAR results?: Humans. AARs propose, humans commit.

From Lesson 15.03, Lesson 15.04, Lesson 15.06, 15.06 code/main.py, 15.07 code/main.py