Experiment Runner
The loop is only as honest as its measurements. Build the runner that takes a spec, executes it in a sandboxed subprocess, and emits a json metrics blob the evaluator can trust. Encode an experiment as a typed spec the runner can serialise to a subprocess. Launch a subprocess with a hard wall clock timeout and a soft memory cap, and surface both as terminal conditions. Capture stdout, stderr, and the structured metrics blob into a single result record. Build an ablation table that sweeps one configuration knob at a time over a fixed base spec. Keep every result deterministic given a seed so the evaluator sees the same numbers across runs. A research loop runs untrusted code. The hypothesis came from a sampler, the experiment script came from the same path; treating either as safe in-process is asking for a crash that takes the orchestrator down. Subprocesses are the simplest isolation the language ships: a separate process, an independent address space, a signal handle on the parent side. The runner here does not implement full sandboxing. There is no cgroup, no seccomp filter, no namespace remapping. What it does have is a wall clock timeout, a polling loop for memory growth, and a kill path that terminates the process on either limit. That is the runtime contract every more…
Experiment Runner: The loop is only as honest as its measurements. Build the runner that takes a spec, executes it in a sandboxed subprocess, and emits a json…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.