Code Exec Metric
Generated code is right when it passes the tests. The eval harness has to extract code, run it without crashing the host, and tally pass-rates honestly. This lesson builds that surface. Extract a code block from a free-form generation in a way that matches the post-process rule from lesson 70. Execute candidate code in an isolated subprocess with a wall-clock timeout, output cap, and an import denylist. Score a task as the fraction of supplied assertion strings that pass against the candidate. Compute pass-at-k for tasks that sample multiple generations from one model. Treat sandbox crashes, syntax errors, and timeouts as first-class fail modes with distinct exit codes the runner can log. Inline exec is a security and stability hazard. A generated while True: pass blocks the eval forever. A generated import shutil; shutil.rmtree('/') is exactly as catastrophic as it sounds. The fix is to spawn a fresh Python interpreter per candidate, pass the code on stdin, write the assertion results to stdout, and kill the process if it overruns. The host eval process keeps running. Real evals like HumanEval, MBPP, BigCodeBench, and LiveCodeBench all use a subprocess sandbox. Some layer Docker on top. We stop at the subprocess for a reason: it is portable, it is stdlib, and it catches the failure modes that matter for educational eval. Production…
Code Exec Metric: Generated code is right when it passes the tests. The eval harness has to extract code, run it without crashing the host, and tally…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.