A prototype that quietly acquired real users
RolesLLM Product EngineeringAgent Systems EngineeringCustomer AI DeploymentInference Engineering
The question
You discover that a weekend prototype you shared internally is now used every day by another team, with production data, and someone just reported an outage to you. Walk me through what went wrong before this moment and what you do in the first week.
Answer it first, aloud or on paper. Then open the answer.
The answer
Show short answer
The system drifted across stages. It gained users, data and authority without gaining ownership, and nothing in the system itself marked it as a prototype. In the first week I make the stage observable in configuration, access control, telemetry and documentation, cut the production data path until a named owner accepts it, and decide with that team whether the right next stage is a bounded pilot or a stop.
Show full answer
Three stages ask three questions
| Stage | Primary question |
|---|---|
| Prototype | Can this mechanism produce the evidence at all? |
| Pilot | Does it work safely with a bounded real audience and real conditions? |
| Production | Can we own it continuously at the promised reliability and risk level? |
These are learning environments, not levels of polish. A prototype can be technically complete and still be disposable. Production begins when the organization accepts ongoing responsibility.
What went wrong
The prototype was built for one question and had no false operational guarantees, which was right. The failure was stage drift: it acquired an audience, real data and the authority to affect another team's work, and nobody acquired ownership. A warning banner would not have prevented this. The stage has to be enforced by the system: a configuration flag that blocks production data, access control that limits the audience, telemetry that shows who uses it, and documentation that says what it is.
The first week
- Stop the bleeding: block the production data path with a technical control, not a message.
- Name the stage in the system so anyone can see it.
- Meet the other team with the pilot requirements: a named audience, a human owner, bounded duration and authority, audit and rollback, outcome and guardrail thresholds, and exit criteria for expand, revise or stop.
- Decide. If no one will own it, the honest outcome is a stop, and the evidence it produced still counts.
What production would require
A service level objective, on-call and incident ownership, security and privacy review, cost and capacity controls, rollback and recovery, continuous monitoring, and a retirement path. Deployment is not production. Ownership is.
Follow-ups
L3SeniorThe other team says the outage shows the tool is valuable. How do you answer?
Show expected answer
value is a reason to run a bounded pilot with an owner, not a reason to keep an unowned system on production data.
L3SeniorWhich technical control would have prevented the drift?
Show expected answer
a configuration or access control that refuses production data unless the stage is set to pilot or production by an owner.
L4StaffWhat is the first operational responsibility that makes a build production?
Show expected answer
someone accepts an on-call or incident ownership commitment; before that, the system has users but no owner.
L4StaffThe team wants to skip the pilot and go straight to production. What evidence would justify that?
Show expected answer
low consequence, operational readiness, and measured outcome and guardrail results from a bounded real audience.
Wrong answers
Show the wrong answers that interviewers hear
“Add a banner that says prototype.”
A banner does not stop data, users or authority from flowing in.
“Harden the prototype into the product since people already use it.”
Prototype code earned another stage only if the evidence says so; usage is not that evidence.
“Shut it down with no discussion.”
The evidence it produced is real; the decision owner needs options, not a surprise.
Taught in
The prototype, pilot or production lesson gives the three stage questions, the pilot requirements and the stage drift controls.