The agent said the tests passed, and no test ran
RolesAgent Systems EngineeringAI Evaluation and ReliabilityCustomer AI Deployment
The question
A pull request from a coding agent says that the whole suite is green. A reviewer runs it and two tests fail. The transcript shows the agent writing "running tests now" and then "all tests pass", with nothing between the two lines. What happened, and what do you build so that this claim cannot reach a reviewer again?
Answer it first, aloud or on paper. Then open the answer.
The answer
Show short answer
The agent predicted the output instead of reading it. Either no command ran, or the command ran and nobody captured the result into the loop. The fix has two parts: a feedback runner that records every command with its exit code, and a verification gate that reads those records and refuses to mark the task done unless each acceptance command ran and exited zero.
Show full answer
Three ways the claim appears
- No command ran. The model wrote the words it expected to write next. Nothing in the transcript shows argv, output or an exit code.
- The command ran, and the result never came back. A runner that errors before it captures the exit leaves the model with nothing to read, and the model fills the gap.
- The result came back, and the failure line was cut. A long output trimmed from the end loses the summary line where the failures are.
The transcript in this case has no record between the two sentences, so it is the first or the second.
The feedback runner
Every command goes through one wrapper. The record carries the exact argv, the tail of stdout and stderr as separate fields, the exit code, the duration, the start time, and a one-line note of what the agent expected. The runner truncates head and tail in a fixed way with a marker for the lines it removed, so the final error line survives. If the runner fails before it captures an exit, the record carries exit_code: null and an error field, and the loop must not advance on it. No exit, no progress.
The verification gate
The gate is a pure function over the artifacts. For this failure it checks three things from the feedback log: every acceptance command appears in the records, each one exited zero, and no record has a null exit. Each of those is a block-severity finding. A block prevents passed: true, and only a human can override it, with a reason and a user id in a signed log. The gate runs in CI and before the handoff. No green verdict, no pull request.
Why the model is not the fix
A prompt that says "always run the tests" asks the model to be honest about something it cannot see. The runner makes the result visible. The gate makes the claim checkable. Neither depends on the model.
Follow-ups
L2WorkingWhy does the record keep stdout and stderr in separate fields?
Show expected answer
a test runner writes the failure summary to one stream and progress to the other, and a merged tail can hide the summary behind progress lines.
L2WorkingThe agent runs the acceptance command, it fails, and a retry passes. What does the gate see?
Show expected answer
two records for the same command, linked by the parent command id. The gate blocks on the failed record and does not report
passed: true, even though the retry exited zero. Only a signed human override gets past it, and the chain keeps the failure history visible.L3SeniorWhy must the gate be deterministic and not a model judge?
Show expected answer
the gate answers whether the task is complete, which must give the same verdict for the same artifacts every time, while a judge gives a qualitative opinion that belongs on the review side.
L4StaffWhere do you place the gate so that it cannot be bypassed?
Show expected answer
in layers, with a pre-commit hook, a CI status check, and a pre-merge gate, so that a failure in one layer is caught by the next.
Wrong answers
Show the wrong answers that interviewers hear
“Tell the agent to paste the test output into the transcript.”
The model can paste what it expects, and a reviewer cannot tell a paste from a capture.
“Trust the agent and let CI catch it.”
CI did catch it, after the claim reached a reviewer, and the point is to stop the claim at the source.
“Treat a timeout as a pass, because the tests were slow.”
A timeout gives a null exit, and the loop must refuse to advance on a null exit.
Taught in
The runtime feedback lesson builds the runner, and the verification gates lesson builds the gate that reads its records.