9.6

Evals in CI: tool calls, rows, and state

Assert on tool calls, database rows, and thread state, and run the suite on each model you deploy, because three real models failed different cases of one suite.

Your 20 conversation tests pass against a scripted model. You point the service at a real model, and seven of them fail. Some failures are the model. Some are the test, which had learned the habits of the scripted model.

When you finish this section, you can write an eval that checks state and not wording. You can tell a model failure from a test failure, and choose which cases block a deploy.

What one eval checks

Eval: one scripted conversation against the running service, followed by assertions on what the service did. The service under test is the real FastAPI app with Postgres. Only the model changes from run to run.

pythonone eval: the refund is proposed, held, and written onceFull log
@case
async def refund_approved_once(c, t, x):
    order = f"ORD-{c.user}-1"
    events = await c.say(t, f"Please refund {order}")
    [pending] = interrupts(events)
    assert (pending["name"], pending["args"]["order_id"], pending["args"]["amount_cents"]) == ("refund_order", order, 12900)
    assert await refunds(x.sql, order) == []
    await c.approve(t)
    assert await refunds(x.sql, order) == [{"amount_cents": 12900}]
    assert finished(await c.state(t))

Every assertion reads something the model cannot word differently. The first reads the interrupt that the stream sent, with the tool name and its arguments. The second and third read the refunds table before and after the approval. The last reads the thread state: no next node, no pending interrupt, and a final AI message with no tool calls.

The other cases follow the same rule. A FAQ case checks that search_faq ran and that its result holds the right entry id. A fault case reads the OpenTelemetry spans of the run, as section 9.4 shows. Two checks read text, and both are literal strings that must not appear: another customer's item name, and another user's door code.

One suite, three models

We measured the 27 tests of the service on the Mac, three times with Qwen3 4B and three times with Qwen3 14B through Ollama. Both ran with temperature 0 and seed 42. Claude Haiku 5.5 ran the 20 evals once. The other seven tests start servers and send many requests, and our budget for Anthropic calls covered the evals only. The backup model was Qwen3 8B for the 4B runs and Qwen3 4B for the others. Raw runs

ModelRunsTests in a runPassed in each run
Qwen3 4B32716, 16, 16
Qwen3 14B32722, 21, 21
Claude Haiku 5.512020

Figure 9.6 puts every test of every run in one grid.

Fig. 9.9the same 27 tests on three real modelschart
The same 27 tests on three real models Each row is one test of the reference service, and each column is one run of the whole suite. Qwen3 4B: 16 of 27, 16 of 27, 16 of 27. Qwen3 14B: 22 of 27, 21 of 27, 21 of 27. Claude Haiku 5.5: 20 of 20. A letter in a failed cell names the cause, with the key below the grid. Qwen3 4B 1 16 2 16 3 16 Qwen3 14B 1 22 2 21 3 21 Haiku 5.5 1 20 run of the suite tests passed in the run faq: return window faq: delivery time order status unknown order other customer order hidden refund approved once a a a g g g refund rejected, no row a a a refund asked twice a a a g g g refund tool replay ticket for a human memory saved, recalled n n n memory of other user flaky faq retried f f f hung tool times out s s s failed ticket not retried primary down, backup used runaway loop, call limit long thread summarized message during approval z z z z z z status, then refund t t t isolation: thread, memory n n n isolation: bad token 401 crash mid refund, resume h h h g g g SIGTERM, run finishes c c SIGTERM past deadline w w w two workers, one thread LangSmith off h h h service tests, not run on Claude a model asked for the amount and the reason, and proposed no refund_order call z model proposed amount_cents 0, the approver approved it, and a 0-cent refund row was written t model opened a ticket with create_ticket in place of a refund_order call n model did not call save_note, so no memory existed to recall f model answered without calling search_faq h test sent an approve after no refund was proposed, got 409, and parsed the 409 body as SSE w test assumed the run lasts more than 3.5 s, and the real run finished before SIGTERM g model invented amount_cents 10000 without reading the order s test bound the whole run to 5 s, and the real model calls took longer c uvicorn closed the stream at its 10 s limit, and the 11.6 to 12.1 s run finished in the drain
Qwen3 4B and Qwen3 14B failed different cases of the same suite, and Claude Haiku 5.5 passed all 20 evals. Each row is one test, and each column is one run of the suite. A plum cell passed. A rose cell failed, and its letter is the cause in the key. A dotted cell did not run.

Why each case failed

We read the stream events, the thread state, and the database rows of every failed case. Each failure has one cause, and the causes fall in two groups.

CauseKindQwen3 4BQwen3 14BClaude Haiku 5.5
asked for the amount and the reason, and proposed no refundmodel3 casesnonenone
invented amount_cents 10000 without reading the ordermodelnone3 casesnone
proposed amount_cents 0, and the approver approved itmodel1 case1 casenone
opened a ticket in place of a refundmodel1 casenonenone
did not call save_notemodel2 casesnonenone
answered without search_faqmodel1 casenonenone
an approve sent after no refund was proposedtest2 casesnonenot run
a time limit that assumed the scripted modeltest1 case1 casenone
uvicorn closed the stream at 10 s, and the run finished in the draintestnone1 case in 2 of 3 runsnot run

Within a model, the causes did not change from run to run. Qwen3 4B failed the same 11 tests in each of 3 runs. Qwen3 14B failed the same four evals and the crash test in each of its 3 runs.

The model failures are the results that the suite exists to find. Qwen3 4B would not call refund_order until the customer gave an amount and a reason. Qwen3 14B called it at once, with an amount that it made up. refund_order refused 10000 cents on an order of 8900, so no money moved. On an order of 12900 cents the same guess would have refunded the wrong amount.

Claude Haiku 5.5 read the order before it proposed a refund, in 5 of 5 refund cases. It read it in the same turn three times and in an earlier turn of the thread twice. It proposed the order total each time.

The zero refund is the case to remember. Both Qwen models proposed amount_cents 0 for Refund ORD-e19-3. The test approved it, as a careless human would, and the tool wrote a refund row of 0 cents. Section 9.3 has the record and the two fixes. With them, the 0-cent row was gone in 6 of 6 reruns, and the refund evals passed no more often than before.

State checks have a blind spot, and Claude Haiku 5.5 found it. After an approved refund was written, it told the customer It's pending human approval, so it isn't final yet. The row existed, the test passed, and the reply was wrong. Read a sample of replies by hand, or add a check for the claims that matter to you.

Tests that knew the scripted model

Three failures were in the tests. The crash test and the LangSmith test asked for a refund and then posted an approve without a check. When the model proposed no refund, /approve answered 409 with a JSON body, and the helper parsed that body as SSE. The test failed with a ValueError that said nothing about the service.

The third test sent SIGTERM 3.5 s into a run and expected the run to still be going. The scripted model slept for 3 s on each call. Qwen3 4B finished the whole run in 1.45 s. The hung_tool_times_out case had the same flaw on Qwen3 14B. It bounded the whole conversation to 5 s, and the run took 11.04 s, while the tool timeout itself worked.

Those tests also leaked. Each one started uvicorn workers and stopped them only on its success path. After the six Qwen runs, 12 workers from failed tests were still running. With the load test's worker, lc_prod had 77 connections, and stopping the 12 left 17. Stop test servers in a finally block, or in a fixture.

So split the suite in two:

  • Service tests check what the service promises: the lock, the owner check, crash recovery, shutdown, and LangSmith staying off. Drive them with a request that leaves the model no choice, such as an explicit amount and reason. Require every run to pass. Section 9.3 and section 9.5 ran them that way.
  • Model evals check what the model does with an ordinary request. Run each one on each model that you deploy, and track a pass rate per case.

Repeats on a local model

Greedy decoding with a fixed seed repeats itself. The three runs of Qwen3 4B passed the same 16 tests. Between the first and second run, 18 of 20 eval transcripts matched word for word. The two that differed were the longest conversations, the runaway loop and the six-turn thread, and their tool calls matched. So three repeats of a greedy local model measure the service and not the model. Vary the seed or the temperature when you want a spread.

The CI job

The file deploy/ci.yml in the app code is a workflow that runs the suite on each push with a Postgres service container. It is not installed under .github/workflows in this repository. As written it uses the scripted model, and we did not run it on GitHub Actions. To run it on a real model, set MODEL and give the job a key or an Ollama service. Keep the two halves apart: the service tests on every push, and the model evals on a schedule with a pass-rate threshold for each case.

LangChain (Python): langchain, langchain-core, langgraph, deepagents, langsmith 1.5.0 · c2f4036 · 2026-11-11 · Edition 2026.11