Outcomes, Workflows, Risks and Specs
Frame the outcome, ground the workflow, rank assumptions, pick a slice, write the spec, set metrics, choose a stage.
- Concepts and terms
- Tasks and commands
- Annotated code
- Diagrams
- Comparisons
- Defaults and numbers
- Troubleshooting
- Questions and tips
From request to stage#
From Lesson 14.47, Lesson 14.48, Lesson 14.49, Lesson 14.50, Lesson 14.51, Lesson 14.52, Lesson 14.53
Choose a stage#
Outcome frame#
- User
- who experiences the problem directly?
- Situation
- when and where does it occur?
- Current behavior
- what happens today, with workarounds?
- Desired outcome
- what observable state improves?
- Constraints
- fixed safety, policy, cost or compatibility limits.
- Non-goals
- tempting adjacent work that is excluded.
- Solution leakage
- a product form, model or framework in the outcome.
Formulas from the labs#
- Risk
- \mathrm{impact} \times \mathrm{uncertainty} + \mathrm{irreversibility}
- Slice score
- \frac{\mathrm{value} + \mathrm{uncertainty\ reduced}}{\mathrm{effort} + \mathrm{risk\ penalty}}
- Risk penalty
- consequence times 2 if not reversible, else times 0.5.
- Eligible slice
- the required proof is a subset of
proves. - Direct evidence ratio
- direct items over all evidence items.
at-most- passes when value is at or below the threshold.
at-least- passes when value is at or above the threshold.
From 14.48 code/main.py, 14.49 code/main.py, 14.50 code/main.py, 14.52 code/main.py
Assumptions and evidence#
- Value
- will the outcome matter enough?
- Usability
- can the user understand and act on it?
- Feasibility
- can the system produce it with the data and constraints?
- Viability
- can the organization sustain cost, ownership and operation?
- Safety
- can it fail without unacceptable consequence?
- Evidence ladder
- direct behavior, artifact, reported behavior, inference.
- Proof of current behavior
- only the first two rungs.
From Lesson 14.49, Lesson 14.48
Three decision modes#
| Factor | Locked | Bounded | Delegated |
|---|---|---|---|
| Agent may | Not choose | Choose inside limits | Own the choice, explain it |
| Use for | Safety, authority, public compatibility | Search budgets, retry counts | Names, reversible refactors |
| Next step | Human checkpoint | Agent proposes within limits | Agent implements and proves |
| Rationale | Required | Required | Optional |
| Output key | human_checkpoint | bounded_decisions | agent_may_decide |
Questions#
- A cheap slice scores high but misses proof?
- It stays ineligible.
- Every result says keep building?
- The slice is not an experiment.
- Can a lower layer prove a higher claim?
- No. A unit test proves a local contract only.
- Offline replay or pilot?
- Neither substitutes for the other.
- When does production begin?
- When the organization accepts ongoing responsibility.
From Lesson 14.50, Lesson 14.51, Lesson 14.52, Lesson 14.53, Lesson 14.48
Tips#
- Avoid false minimums: UI-only, infrastructure-only, happy-path, demo, platform.
- Write a stop rule before implementation.
- Write pass, fail and ambiguous paths before you see results.
- Mark the stage in configuration, access control, telemetry and docs.
Prototype, pilot, production#
| Factor | Prototype | Pilot | Production |
|---|---|---|---|
| Question | Can the mechanism produce the evidence at all? | Does it work safely with a bounded real audience? | Can we own it at the promised reliability? |
| Use when | No real users or data needed | Real behavior needed, consequence not ready | Consequence bounded, operations ready |
| Controls | Synthetic or recorded inputs, discardable | Limited audience, rollback, human owner, audit trail | SLO, on-call owner, security review, monitoring |
| Exit | Mechanism earns another stage | Expand, revise or stop | Retirement path |
Metric contract#
plan = MeasurementPlan( goal="Reduce time to identify the affected service without increasing unsafe actions", questions=["How quickly is the correct service identified?"], metrics=[ Metric("median_identification_seconds", "at-most", 120, "ten incident replays", "replay log", "outcome"), Metric("correct_service_rate", "at-least", 0.9, "ten incident replays", "incident record", "outcome"), Metric("production_writes", "at-most", 0, "entire pilot", "audit log", "guardrail"),required kind ])report(plan, values)missing value gives status missingLab functions#
| Task | Code |
|---|---|
| Run a lab and its tests | python3 -m unittest discover code/ |
| Frame status | decision(frame) |
| Audit workflow evidence | audit(steps) |
| Rank assumptions | prioritize(items) |
| Highest open risk | next_experiment(items) |
| Pick an eligible slice | choose(items, |
| Spec to contract | compile_contract(specification) |
| Stage and its controls | plan(decision) |
From 14.47 code/main.py, 14.48 code/main.py, 14.49 code/main.py, 14.50 code/main.py, 14.51 code/main.py, Lesson 14.53, 14.53 code/main.py
Lab constants#
| Setting | Default | Note |
|---|---|---|
| Risk dimensions | range(1, 6) | Impact, uncertainty, irreversibility |
Evidence confidence | 0 <= item.confidence <= 1 | Else an issue |
Workflow order | range(1, len(steps) + 1) | Contiguous from one |
VALID_MODES | {"locked", "bounded", "delegated"} | Other modes are invalid |
Stage consequence | 1 <= decision.consequence <= 5 | Else an error |
| Pilot at consequence | 4 | Or not reversible, or not ready |
From 14.48 code/main.py, 14.49 code/main.py, 14.51 code/main.py, 14.53 code/main.py
Validator messages#
| Symptom | Cause | Fix |
|---|---|---|
desired outcome names the proposed output | Solution leaked into the outcome | State the observable result instead |
non-goals are empty | No boundary on the slice | Add concrete non-goals that reject work |
no slice proves the required assumptions | Candidates skip the risky part | Design a slice that covers the proof set |
constrained decision lacks rationale | Locked or bounded without a reason | Write the reason for the limit |
guardrail metric is missing | Only outcome metrics | Add a production_writes guardrail |
From 14.47 code/main.py, 14.50 code/main.py, 14.51 code/main.py, 14.52 code/main.py