9.5

ToolRetryMiddleware, ModelFallbackMiddleware, and shutdown

Time-limit every tool, retry only the tools that are safe to repeat, and expect the model to decide what the customer hears about a fault.

A help-center search fails once with a connection error. An order lookup hangs. The ticket system returns an error, and you cannot tell whether it wrote a row. The primary model provider is down. Each of these needs a different response, and the middleware stack of create_agent is where you choose it.

When you finish this section, you can order the middleware so that each tool attempt has its own time limit. You can choose which tools to retry, add a backup model, and shut a worker down without losing a run.

The stack, from the outside in

pythonthe middleware stack of the serviceFull log
        middleware=[
            SummarizationMiddleware(model, trigger=("messages", settings.summarize_after), keep=("messages", 6)),
            ModelCallLimitMiddleware(run_limit=settings.model_calls_per_run, exit_behavior="end"),
            ModelFallbackMiddleware(backup),
            HumanInTheLoopMiddleware(
                {"refund_order": {"allowed_decisions": ["approve", "reject"], "description": refund_description}},
                interrupt_mode="per_call",
            ),
            ToolErrorMiddleware(tool_error_message),
            ToolRetryMiddleware(tools=IDEMPOTENT, max_retries=2, initial_delay=0.1, on_failure="error"),
            Tracing(),
            ToolTimeout(settings.tool_timeout, {"refund_order": settings.refund_timeout}),
            *extra,
        ],

The first middleware in the list wraps all the others. For a tool call, the order from the outside in is the approval, the error text, the retry, the span, the timeout, and the tool. Three results follow from that order.

  • Each attempt gets its own time limit, because the timeout is inside the retry.
  • Each attempt gets its own span, because the span is inside the retry too.
  • The model sees one error message after the last attempt, because ToolErrorMiddleware is outside the retry.

ToolTimeout is ours, and it is four lines of asyncio.wait_for. In LangChain 1.5.0 only the shell and file search middleware set time limits, and a general tool call has none. refund_order gets 15 s and the other tools 5 s. The tests set 0.5 s.

A timeout cancels the wait for the tool. It does not undo the effect. A refund tool can commit its row before the limit fires. After a timeout on a tool that moves money, read the order before you tell the customer that nothing happened.

Idempotent tool: a tool that gives the same result and the same side effects when you call it twice with the same arguments. Only read_order, search_faq, and recall_notes are on the retry list. create_ticket and save_note write a row with a new id on each call, so a retry after a lost reply would write two. refund_order has its own key, which section 9.3 explains.

The error text tells the model what to do with the failure.

pythonthe text that the model reads after a failed toolFull log
def tool_error_message(exc, request):
    event("tool.error", tool=request.tool_call["name"], error=type(exc).__name__)
    if isinstance(exc, TimeoutError):
        return f"{request.tool_call['name']} timed out. Tell the customer and do not guess."
    return f"{request.tool_call['name']} failed with {type(exc).__name__}. Tell the customer and do not guess."

Five faults on three models

The eval suite injects each fault with a test middleware that sits inside the timeout, where the tool would be. We measured each fault once per model on the Mac, in the first run of the suite. The service did the same thing on every model. The models differed in whether they called the tool at all, and in what they told the customer.

FaultWhat the service did, from the spansQwen3 4BQwen3 14BClaude Haiku 5.5
search_faq fails once2 attempts, the first an error spandid not call search_faq2 attempts, then the right answer2 attempts, then the right answer
read_order hangs3 attempts, each cut at about 0.50 s3 attempts, then told the customer that the order was not found3 attempts, then said it could not look the order up now3 attempts, then said the order system timed out
create_ticket fails1 attempt, no retry, 0 ticket rowssaid it could not open a ticketsaid it could not open a ticketsaid it could not open a ticket
primary model downeach primary call an error span, then the backup answeredbackup Qwen3 8B, 2 callsbackup Qwen3 4B, 2 callsbackup Qwen3 4B, 2 calls
model asks for search_faq foreverstopped at 8 model calls and 8 tool callsrun took 29.61 srun took 151.34 srun took 20.89 s

Figure 9.5 lays the same runs out as one decision table.

Fig. 9.8five faults, what the middleware did, and what each model saiddecision
Five faults, the attempts in the spans, and what each model told the customer For each injected fault, the tool and model spans of the first suite run on the Mac, and the reply of Qwen3 4B, Qwen3 14B, Claude Haiku 5.5. search_faq fails once: Qwen3 4B no tool call, said no data bad, Qwen3 14B answered after the retry ok, Claude Haiku 5.5 answered after the retry ok. read_order hangs every attempt: Qwen3 4B said order not found bad, Qwen3 14B said it cannot look it up now ok, Claude Haiku 5.5 said lookup timed out ok. create_ticket fails once: Qwen3 4B said it could not open one ok, Qwen3 14B said it could not open one ok, Claude Haiku 5.5 said it could not open one ok. primary model raises: Qwen3 4B answered from backup ok, Qwen3 14B answered from backup ok, Claude Haiku 5.5 answered from backup ok. model asks for search_faq forever: Qwen3 4B answered after 8 calls ok, Qwen3 14B answered after 8 calls ok, Claude Haiku 5.5 answered after 8 calls ok. FAULT SPANS OF THE RUN Qwen3 4B Qwen3 14B Haiku 5.5 tool ok error model call ok search_faq fails once 2 tool spans, Qwen3 14B no tool call, said no data answered after the retry answered after the retry read_order hangs every attempt 3 tool spans, Qwen3 4B said order not found said it cannot look it up now said lookup timed out create_ticket fails once 1 tool span, Qwen3 4B said it could not open one said it could not open one said it could not open one primary model raises model.call: primary, backup answered from backup answered from backup answered from backup model asks for search_faq forever 8 tool spans, Qwen3 4B answered after 8 calls answered after 8 calls answered after 8 calls
The middleware handled each fault the same way on every model, and only the reply to the customer differed. Read each row left to right: the fault, the attempts from the spans, and the reply of each model. A rose reply misled the customer or skipped the tool.

The row to remember is the hung order lookup on Qwen3 4B. The tool message said read_order timed out. Tell the customer and do not guess. The model told the customer I couldn't find your order. Please check the order ID and try again. That reply is false, and it sends the customer to fix an order id that was right. No middleware can stop it. Check the reply to a timeout in your evals, and choose a model that passes that check.

The backup model

ModelFallbackMiddleware tries the next model when a call raises. We measured the primary outage on the Mac: every primary call raised a ConnectionError and left an error span, and the backup made the call. The backup ran on each model call of the turn, so the primary was tried and failed twice. Nothing remembers that the primary is down. A circuit breaker that skips a failed provider for a while is code that you add.

A backup on another provider is an option. For the Claude Haiku 5.5 runs the backup was Qwen3 4B on the Mac.

The same Haiku run had one fault that we did not inject. While the backup model loaded, one read_order attempt took 0.70 s and hit the 0.5 s limit of the tests. The retry returned in 0.06 s, and the customer got the order status. So the retry on an idempotent tool also covered an ordinary slow moment of the machine.

A loop that does not end

ModelCallLimitMiddleware with run_limit=8 and exit_behavior="end" ended the runaway runs after the eighth model call, on every model. The run ended with status done and a final AI message, so the client saw a normal end. The limit counts model calls in one run, so set it from the longest honest task in your evals.

Shutdown

Uvicorn waits for open connections when it gets SIGTERM, up to --timeout-graceful-shutdown. Then the FastAPI lifespan exits, and the service waits up to DRAIN_SECONDS for each run task and cancels the rest. A cancelled run keeps the checkpoints that durability="sync" already wrote.

We measured both cases with Qwen3 14B on the Mac, five trials each. Each trial asked a FAQ question, sent SIGTERM to the worker during the run, and read the thread from a new worker. Raw runs

CaseSettingsClientThread after the restart
finishSIGTERM at 0.3 s, uvicorn 20 s, drain 20 sdone in 8.28 to 9.16 s in 4 of 5 trialsanswered in 5 of 5
cancelSIGTERM at 0.3 s, uvicorn 1 s, drain 0.05 sstream closed with an error in 5 of 5next was ["model"], with the human message only, in 5 of 5

In the first finish trial, the first model call took 14.96 s, because Qwen3 14B loaded after the 4B trials. At 20 s uvicorn logged Cancel 1 running task(s), timeout graceful shutdown exceeded and closed the stream. The run task went on, and it finished during the drain at 21.58 s. The thread had its answer, and the client never saw done. The suite showed the same thing twice with a 10 s limit.

The Mac was loaded for these trials, with a load average of 15.6 to 28.7, so read the seconds as upper bounds. The counts are the result.

After each cancel, a new worker resumed the thread with /approve and no decision. The run finished in 7.27 to 9.10 s and answered from the FAQ, in 5 of 5 trials.

So a run outlives its stream when the uvicorn timer is shorter than the drain timer. Set the uvicorn graceful timeout above your longest run, and tell clients to read the thread state after a dropped stream.

LangChain (Python): langchain, langchain-core, langgraph, deepagents, langsmith 1.5.0 · c2f4036 · 2026-11-11 · Edition 2026.11