Turn telemetry nobody owns into a learning loop
RolesAI Evaluation and ReliabilityLLM Product EngineeringAgent Systems EngineeringCustomer AI Deployment
The question
You inherit a team that collects traces, evaluation scores, support tickets and incident logs for an assistant. Nobody can name a change that any of it caused. Design the mechanism that turns those signals into improvements, and tell me how you keep it from becoming another backlog.
Answer it first, aloud or on paper. Then open the answer.
The answer
Show short answer
The missing piece is promotion: a defined path from an observation to a durable change with an owner and proof. Each signal is connected to an outcome, a constraint or an assumption, traced to a root cause, and routed to the earliest layer that owns that cause, which is evaluation, context, policy, runtime or a shaped backlog item. Every resulting action carries one owner, a priority from consequence and recurrence, the artifact to change, the verification, a review window and a retirement condition, so the loop produces controls that are later re-examined rather than rules that only accumulate.
Show full answer
The loop
- Observe a concrete signal.
- Connect it to an outcome, a constraint or an assumption.
- Identify the earliest system layer that owns the cause.
- Create a bounded change.
- Verify that recurrence became less likely.
- Review whether the control should remain.
Route to the owning layer
| Signal | Destination |
|---|---|
| False positive, regression, wrong result | Evaluation or test |
| Missing context, duplicate work, stale fact | Context source or retrieval route |
| Unsafe action or authority gap | Policy or permission boundary |
| Timeout, retry storm, unavailable dependency | Runtime control |
| New product need or unresolved tradeoff | Shaped backlog item |
A timeout belongs to the runtime, not to the general backlog, and a wrong answer belongs to a test, not to another prompt paragraph. The cause is fixed at the earliest effective layer.
Ownership is part of the control
An unowned improvement is an observation with better formatting. Every action gets one owner, a priority based on consequence and recurrence, the artifact to change, the verification that proves the change, a review or expiry window, and a retirement condition.
Why it is not a backlog
A backlog stores wishes. The ratchet stores controls with proof and expiry. Controls are reviewed when the architecture changes, when a lower-level invariant replaces a higher-level instruction, when the protected failure has stayed away across the window, or when the control blocks legitimate work more than it prevents harm. Retirement needs evidence too.
Metrics close the loop
The signals only mean something against metrics defined before the result: outcome, guardrail and counter-metric, each with a threshold, a window and a source. The ratchet then changes the outcome frame, the assumptions, the slice or the measurement plan for product evidence, and the tests, context, scope or handoff for coding-agent corrections. Incidents can change both.
Follow-ups
L3SeniorTwo teams share the assistant and disagree on a tradeoff a signal exposes. Where does it go?
Show expected answer
to a shaped backlog item with a human decision owner; the ratchet does not resolve product disagreements by itself.
L4StaffHow do you prioritize across many signals?
Show expected answer
by consequence and recurrence together, so a severe single event and a frequent minor one both rise above a rare minor one.
L4StaffWhat proves the loop works after a quarter?
Show expected answer
named controls that trace back to signals, verification results that show recurrence fell, and retirements with their evidence; not the volume of telemetry collected.
Wrong answers
Show the wrong answers that interviewers hear
“Build a dashboard so everyone can see the signals.”
Visibility without promotion and ownership changes nothing.
“Add each lesson learned to the system prompt.”
Prose is the weakest layer; a permission boundary can make the unsafe action impossible, and a test can block a release that repeats the failure.
“Keep every control forever since it was earned.”
Controls decay and conflict; each one needs a retirement condition and periodic review.
Taught in
The feedback ratchet lesson gives the routing table and the ownership fields, and the success metrics lesson gives the metric contract the signals are judged against.