Turn be careful into a rule a checker can score
RolesAgent Systems EngineeringCustomer AI DeploymentAI Evaluation and Reliability
The question
An instructions file tells the agent to be careful, to test thoroughly, and to ask if it is unsure. Three days later the agent ships a change with no tests, writes into a directory it should not touch, and never asks a question. Redesign these instructions so that a runtime can enforce them and a reviewer can score them.
Answer it first, aloud or on paper. Then open the answer.
The answer
Show short answer
Rewrite each wish as a rule with a slug, a category, a one-line description and a named check function. Five categories cover most rules: startup, forbidden, definition of done, uncertainty and approval. Give every rule a severity of block, warn or info, so that the runtime refuses only on block. Keep the rules in one markdown file with one rule per heading, so that review can see every change.
Show full answer
From wishes to rules
| Wish | Category | Rule | Check |
|---|---|---|---|
| Be careful | Forbidden | Never edit the release script or production config | A diff check against forbidden globs |
| Test thoroughly | Definition of done | The acceptance command exits zero, and the feedback log holds the record | A read of the feedback records |
| Ask if unsure | Uncertainty | Open a question note instead of guessing when an assumption is not backed by a file | A check that a question note exists when the state file lists an assumption with no source |
| Be careful | Approval | Any new dependency or any production write needs a human sign-off | A check that the approval record exists before the action |
| Be careful | Startup | The state file exists and is fresh before work begins | A file check at session start |
A rule that does not fit one of the five categories usually wants to be two rules. Force the split.
Severity at write time
Each rule carries block, warn or info. The checker reports all three. The runtime refuses only on block. Tagging at write time forces the calibration up front, because teams that start with every rule as a hard rule later weaken them under deadline pressure without a record. An override of a block rule goes into a signed audit log through the verification gate.
Expiry as a forcing function
Every rule carries an expiry date, 90 days from authoring by default. The checker warns when a rule has had zero violations for 60 days in a row. At the next review the team keeps it with a reason, lowers it to info, or deletes it. One large review deployment in the lesson, with 131,246 review runs across 5,169 repositories in 30 days, found that rule sets with explicit expiry stayed under 30 rules per repository, while sets without expiry grew past 80 rules that mostly never fired.
Source and cache
The markdown file is the authored source. A JSON lock file is the cache that the checker reads on the hot path, regenerated by a pre-commit hook. The markdown diff is reviewable, and the parse stays out of every turn.
Rules and runtime guardrails
Framework guardrails enforce rules during a turn. The rule file is the human-readable contract that those guardrails implement, and the checker proves in CI that the runtime did the right thing. You need both.
Follow-ups
L2WorkingWhy delete a stale rule instead of commenting it out?
Show expected answer
the workbench is the source of truth, and a commented rule is a rule the agent might still read while the checker ignores it.
L3SeniorA rule says to follow a published style guide. Why does it fail the design?
Show expected answer
it has no enforcement command, so the agent can invent compliance, and the fix is to pair it with the exact lint command.
L3SeniorWhere does the uncertainty check get its signal?
Show expected answer
from the state file, which lists assumptions, and from the question notes, so an assumption with no source and no note is a violation.
L4StaffHow does the rule set stay portable across three agent products?
Show expected answer
the file is markdown plus function names, so each runtime registers the same checks as its own guardrails or interrupts while the file stays the contract.
Wrong answers
Show the wrong answers that interviewers hear
“Make every rule a hard block so that the agent takes them seriously.”
Teams quietly weaken blocks under deadline pressure, and a rule set where everything blocks ends with nothing enforced.
“Keep the rules in the prompt, where the model reads them.”
The model reads them and still cannot check them, while a named check function gives the runtime and the reviewer the same verdict.
“Rules are permanent because they came from incidents.”
A rule that never fires still costs attention, so expiry forces a review that keeps, lowers, or deletes it with a reason.
Taught in
The executable constraints lesson parses the rule file into checks and scores a run against it.