Multi-Agent Case Studies and Eval Farms

Three 2026 reference systems, the software team capstone, and a fenced, resumable eval farm in Go.

Concepts and terms
Tasks and commands
Annotated code
Diagrams
Comparisons
Defaults and numbers
Troubleshooting
Questions and tips

Three reference cases#

Factor	Anthropic Research	MetaGPT / ChatDev	OpenClaw / Moltbook
Topology	Supervisor-worker	SOP role decomposition	Population-scale substrate
Key result	+90.2% over single-agent Opus 4	MacNet: >1000 agents via DAGs	~2.3M agent accounts in days
Cost signal	15x tokens per query	Per-role budget	Per-interaction pricing
Pattern to copy	Fresh-context subagents	Structured artifact handoffs	Prompt-injection threat model
Verification	Explicit verifier role	QA Engineer	In-network validators
Framework	Claude Agent SDK or LangGraph	CrewAI or MetaGPT	Custom substrate + MCP + A2A
Pick for	Research and knowledge tasks	Engineering tool chains	Network-effect social products

From Lesson 16.25, 16.25 code/main.py

Lease and receipt rules#

Acquire: a done or live lease is a conflict

if l.Done || (l.Worker != "" && now < l.Until) {

l.Until = now + ttl

l.Version++

Submit: check owner and version first

if l.Worker != worker || l.Version != version {

a done lease accepts only the same result again

if l.Result == result {

a first completion must come before expiry

if now >= l.Until {

From solution/stage2.go, solution/stage3.go

Defaults and limits#

Setting	Default	Note
`--shards`	4	Fixed partition count, range 1..1024
`--workers`	2	Concurrent processes, range 1..64
`--lease-ms`	30000	Claim duration
`--now-ms`	-1	Uses the wall clock
`--delay-ms`	0	Range 0..60000
`--store`	farm-data	Same-host directory
Snapshot size	4 MiB	Whole-run snapshot
Result size	1 MiB	Largest accepted result

From solution/main.go, distributed-eval-farm/API.md, distributed-eval-farm/README.md, docs/en.md

Failure modes#

Symptom	Cause	Fix
Worker exits 86	Intentional crash after claim	Run again at the expiry boundary
Run exits 1 with complete:false	Remaining leases still owned	Invoke run again after expiry
Old worker gets a fencing error	Replacement committed first	Expected. Snapshot stays unchanged
Run exits 1 on a changed dataset	Dataset hash differs	Use a different store
Subtasks cannot be implemented	Plan too vague	Plan template with explicit interfaces
Reviewer approves a bug	Reviewer false-approve	Injected-bug probe, tune the prompt
Viral prompt spreads to agents	Prompt injection at scale	Treat cross-agent messages as untrusted
Research agents hallucinate	No explicit verifier role	Add a verifier, about 20-30% of tokens

From distributed-eval-farm/README.md, distributed-eval-farm/API.md, docs/en.md, Lesson 19.10, Lesson 16.25

Common patterns#

Orchestrator + workers: every case has a supervisor or network effect.
Handoff contracts: artifacts between roles follow a schema.
Verification: a first-class role in every case.
Scaling: topology + substrate, not just more agents.
Effort scaling: simple is 1 agent with 3-10 tool calls. Complex is 10+ subagents.
Rainbow deploy: keep old runtime versions alive for in-flight agents.
Prompt injection: treat agent profiles and cross-agent messages as untrusted input.

From Lesson 16.25

Map a design to a case#

def map_to_case(d: Design) -> str:

if d.task_type == "population" or d.user_facing_network:

return "openclaw_moltbook"

if d.task_type == "engineering" or d.roles_distinct:

return "metagpt_chatdev"

return "anthropic_research"research and the default

print_case(case_key)patterns, framework, citation

print_landscape()FRAMEWORK_LANDSCAPE rows

From 16.25 code/main.py

Software team build steps#

From Lesson 19.10

Eval farm architecture#

From distributed-eval-farm/README.md, distributed-eval-farm/API.md, docs/en.md

Eval farm commands#

Command	What it does
`go build -o /tmp/eval-farm .`	Build the reference binary in `solution`
`init --store DIR --input FILE --shards N`	Bind dataset hash and shard count
`run ... --shards N --workers W`	Launch bounded process waves
`worker --store DIR --worker-id NAME`	Claim, evaluate and submit one shard
`status --store DIR`	Validate and report the snapshot
`demo --input FILE`	Print crash and recovery snapshots
`--crash-after-claim`	Exit 86 after the durable claim
`--all --solution --strict`	Grade the reference with `project_test.py`

From distributed-eval-farm/README.md, distributed-eval-farm/API.md, docs/en.md

Eval farm contracts#

Task	Code
Stage 1, stable shard per ID	`Partition(ids []string, shards int)`
Hash an ID	`h := fnv.New32a()`
Stage 2, fence a lease	`Acquire(l, worker, now, ttl)`
Stage 3, accept one result	`Submit(l, worker, version, now, result)`
Stage 4, bounded pool	`Parallel(ctx, ids, workers, run)`
Lease fields	`Shard, Worker, Version, Until, Result, Done`
Errors	`ErrInvalid`, `ErrConflict`, `ErrLimit`

From solution/stage1.go, solution/stage2.go, solution/stage3.go, solution/stage4.go, solution/types.go

Numbers to remember#

Token usage: explains 80% of BrowseComp variance.
MAST: failure rates of 41-86.7% on real multi-agent systems.
OpenClaw: 247k GitHub stars in ~4 months.
Capstone eval: 50 SWE-bench Pro issues against one Sonnet 4.7 baseline.
Rubric: 25 pass@1, 20 speedup, 20 review quality, 20 tokens, 15 coordination.
Farm sample: four records score 3/4. Two waves launch four processes.

From Lesson 16.25, Lesson 19.10, distributed-eval-farm/README.md

Questions#

Why does multi-agent research win?: Each subagent gets a fresh context window.
Which benchmark?: SWE-bench Pro. Verified is contaminated.
What does the capstone measure?: Whether multi-agent wins per dollar, not only whether it works.
Can a late correct worker overwrite?: No. old/v1 fails when new/v2 completed the shard.
Does recovery at 109 steal the lease?: No. It reports a pending run and exits 1.

From Lesson 16.25, Lesson 19.10, distributed-eval-farm/README.md, docs/en.md