Cheat sheets

Research Agent Pipeline from Scratch

Generate hypotheses, retrieve papers, run sandboxed experiments, judge results, and publish a cited report.

  • Concepts and terms
  • Tasks and commands
  • Annotated code
  • Diagrams
  • Defaults and numbers
  • Troubleshooting
  • Questions and tips

Source

The research loop#

queue headnot settledExperimentResultverdictbest branchdraftmodelHypothesisGeneratorlesson 50,ranked queuestoreRetrievalClientlesson 51, BM25+ citation graphprocessExperimentRunnerlesson 52,subprocesslimitsprocessEvaluatorlesson 53,paired t testclientSeed prompttemperature rampprocessTree searchnovelty,quality, budgetmodelLaTeX writervision critiquemodelReviewerensemble5 judgesqueue headnot settledExperimentResultverdictbest branchdraftmodelHypothesisGeneratorlesson 50, ranked queuestoreRetrievalClientlesson 51, BM25 +citation graphprocessExperimentRunnerlesson 52, subprocesslimitsprocessEvaluatorlesson 53, paired ttestclientSeed prompttemperature rampprocessTree searchnovelty, quality,budgetmodelLaTeX writervision critiquemodelReviewer ensemble5 judges

Run it#

CommandWhat it does
python3 code/main.pyLessons 50 to 53 demo. Prints JSON
python main.pyLesson 05 tree search, budget $30
docker run --network=none --memory=8g --cpus=2 --pids-limit=256 --read-onlySandbox for one experiment
run_report.py "<question>" --out out/ --code <dir>Report, trace and evidence JSON
run_report.py --eval heldout/questions.jsonScore the public fixtures
run_report.py "<question>" --model replay --cassette <file>Plan from recorded replies
run_report.py "<question>" --corpus <dir> --compare <old>/report.jsonWrite changes.json
project_test.py research-report-agent --stage <n> --path <dir>Grade a stage and all earlier ones

Loop calls in code#

TaskCode
Configure the rampGeneratorConfig(n_passes=4, t_min=0.2, t_max=1.1)
Draft and rankqueue, logs = generator.run(seed_prompt)
Build a retrieverbuild_client(RetrievalConfig(top_k_lexical=5, max_hops=2))
Search papersresult = client.search(query)
Run one specExperimentRunner().run(spec)
Sweep one knobAblationRunner(runner).sweep(base, "k", [4, 8, 16, 32])
Pick the metricMetricSpec(name="perplexity", direction=LOWER, scale=LOG)
Judge the runsevaluator.evaluate(1, metric_spec, candidates, baselines)

Spec a sandboxed experiment#

base = ExperimentSpec(
spec_id="demo_base", hypothesis_id=1,
script_path=script,prints one json line
config={"k": 16, "steps": 4, "sleep_s": 0.0},
seed=7,sent as config["__seed"]
wall_timeout_s=15.0,hard, killed on exceed
memory_cap_mb=256,soft, polled RSS
metric_keys=["perplexity", "final_loss"],
)
runner = ExperimentRunner()
base_result = runner.run(base)terminal: ok, timeout, oom, crash
base_result.to_dict()metrics, peak_rss_mb, stderr_tail
ablate(base, "k", [4, 8])spec_id demo_base_k_4

Loop defaults#

SettingDefaultNote
n_passes6t_min 0.2 to t_max 1.2
novelty_threshold0.25Lower distance is a duplicate
BM25 k1, b1.5b is 0.75
top_k_lexical10Seeds for the graph walk
max_hops2Values above 2 are clamped
wall_timeout_s30.0memory_cap_mb is 512
improvement_threshold0.02Smaller change is noise
significance_threshold0.05Larger p value is noise

Report agent run#

1indexBM25Index(documents) ranks the markdown corpusin Rust.2planplan_research splits the question into up to 4facets.3gathergather_snippets keeps 3 snippets per facet withoffsets.4writewrite_report composes sentences with [S1]markers.5verifyreview scores each sentence. apply_verdictsdrops the rest.6statedecide_state gives completed, needs_review orfailed.7publishwrite_outputs writes report.html, report.jsonand trace.json.

Scores and formulas#

Cosine distance
1 - dot(a, b) on unit vectors.
rank_score
0.4 novelty + 0.3 specificity + 0.3 testability.
final_score
0.5 normalized BM25 + 0.3 graph + 0.2 recency.
graph_score
1 for a lexical hit, 0.6 at one hop, 0.3 at two hops.
Node score
novelty 0.4 + quality 0.5 + budget weight 0.1.
Improvement
(baseline - candidate) / abs(baseline) when lower is better.
Paired t
t_stat = mean / sqrt(variance / n), df is n - 1.
Report score
100 x (0.4 precision + 0.3 recall + 0.3 facts).

Report agent defaults#

SettingDefaultNote
--modelrulesAlso replay or live
Budget max_steps50BudgetExceeded gives failed
Budget max_tokens20000estimate_tokens is len / 4
review threshold0.6Overlap needed to keep a sentence
citation_precision threshold0.8Stricter than the critic
max_facets4Overview, how, risks, when
max_sentences3Per section
Reference score87.5Six public questions, out of 100

Failures and fixes#

SymptomCauseFix
Log shows parse:no hypothesis block foundReply has no tagged blockKeep temperature at or below 1.2
Every pass logs duplicateDrafts share most nounsWiden the seed prompt and run again
terminal is crash with metricsNon-zero exit codeFix the script. Partial runs count as failed
PairingErrorSeeds differ between the listsRerun both arms on the same seeds
Verdict noise with a big gainFewer than 2 pairs, no p valueRun more seeds
Node fails loss_divergedLoss above 4.0Drop that branch. Expand others
Sentence dropped: names or numbers not in sourceSwapped name or changed numberKeep writing extractive
Plan source is rules-fallbackBad model JSON or cassette missRead the plan note. Rerecord the cassette

Questions#

Why ramp the temperature?
Later drafts wander. Below 0.2 the model parrots the seed.
Why walk the citation graph?
It finds papers that use other words than the query.
Why only two hops?
Three hops blows up the result and drifts off topic.
Why a subprocess?
Untrusted code must not crash the orchestrator.
Why one knob at a time?
Full factorial sweeps grow exponentially.
Why a paired test?
Each seed gives a difference. Noise floor is their standard error.
Is needs_review a failure?
No. It tells a human exactly where to look.