Two hundred registered tools, forty visible at a time
RolesAgent Systems EngineeringCustomer AI Deployment
The question
Your harness has grown to about two hundred tools across several teams. The model cannot hold all of them in context, two teams once registered the same name and the second silently replaced the first, and the success rate has fallen as the catalog grew. Design the registry and the policy for what the model sees on each turn.
Answer it first, aloud or on paper. Then open the answer.
The answer
Show short answer
Make the registry the only source of truth for what exists, what shape the arguments take and which handler runs, and surface only ten to forty tools per turn. The registry holds a typed record per tool, rejects a second registration of a name unless the caller asks to override, and validates every call against a small schema subset with pointer-shaped errors. The exposure policy loads tool metadata into context only when the task needs it, and the fewer tools the model sees, the better it calls them.
Show full answer
The record
| Field | Purpose |
|---|---|
name | Unique, lower-case segments separated by dots |
description | One line shown to the model |
schema | A subset of JSON Schema with eight keywords |
handler | The callable, opaque to the validator |
idempotent | Lets the dispatcher decide whether a retry is safe |
timeout_ms | Per-tool override of the dispatcher default |
The schema is data and the handler is code. Keep them apart so that validation never ends up inside handlers.
Three registry rules
- No silent overwrite.
registerraises on a known name unlessoverride=True. The incident in the prompt is this rule missing. - Validate before dispatch. A pure, recursive walk over the schema returns
Okor a list of errors with JSON pointer paths. No coercion, no handler call, no I/O, so the validator can run again over a replay log. - The registry is shape, not safety. Timeouts and sandboxing live in the dispatcher. A validated call can still misbehave.
The exposure policy
A catalog of two hundred tools and a context that fits forty is a routing problem. Surface a subset per turn based on the task, and load the rest just in time. The workbench lesson describes the pattern as skills that are revealed step by step: a function registry whose metadata enters context only when needed. The same lesson reports a team that deleted 80 percent of its agent's tools and saw the success rate go from 80 percent to 100 percent. More tools in view means more wrong choices, longer prompts and more schema to misread.
Practical selection signals: the active scope contract, the task type on the board, the tools used in the last few turns, and an explicit request from the model for a named family. Log the exposed set per turn so that a failure can be traced to a missing tool rather than a wrong call.
What grows next
Past about fifty tools, add $ref against a local definitions block so that shared argument shapes are written once, and additionalProperties: false so that an extra argument is an error rather than a silent drop. Both are small. Both are common.
Follow-ups
L3SeniorTwo teams want the same tool name for different handlers. What do you do?
Show expected answer
refuse the collision, namespace by dotted segment, and let the exposure policy pick the family the task needs.
L3SeniorHow do you prove the validator is still correct after a refactor?
Show expected answer
run it over a saved replay log of real calls, which works because it has no I/O, no clock and no globals.
L4StaffWhat breaks if the exposure policy is itself a model call?
Show expected answer
a second nondeterministic step on every turn, so the selection should be rule-based from the contract and the board, with the model allowed only to request a named family.
L4StaffThe dispatcher retries a failed call. Which record field decides whether it may?
Show expected answer
idempotent, because a retry of a non-idempotent tool can repeat a side effect.
Wrong answers
Show the wrong answers that interviewers hear
“Give the model the whole catalog and let it choose.”
The model cannot hold two hundred schemas, and success fell as the catalog grew, so the fix is fewer tools in view.
“Put validation in each handler, close to the code it protects.”
Validation inside handlers drifts per team and turns the dispatcher into a guessing game that ends in a stack trace.
“Allow re-registration so that teams can hot-fix a tool.”
A silent replace is the incident in the prompt, and a fix should pass an explicit override that shows up in review.
Taught in
The tool registry lesson builds the record and the validator, and the why capable models still fail lesson places skills and tool catalogs among the harness primitives.