Cheat sheets

Decision, Change and Tutorial Tools

Four evidence tools. Score options, brief page changes, check API tutorials and compare local model runs.

  • Concepts and terms
  • Tasks and commands
  • Annotated code
  • Diagrams
  • Comparisons
  • Defaults and numbers
  • Troubleshooting
  • Questions and tips

Source

Four projects side by side#

FactorDecision Tradeoff ExplorerWeb Change BriefAPI Tutorial RunnerLocal Model Eval Harness
LanguageNode 22.18+Go 1.22 or newerNode 22.18 or newerPython 3.10+
Inputcriteria, optionstwo HTML snapshots, or baseline and URLtutorial, fixture or endpointmanifest, source, labels, records
Outputscriteria.csv, decision-record.json, index.htmlindex.html, changes.json, baseline.jsonrequests.json, results.json, junit.xml, report.htmlscorecard.json, reliability.html
GateChosen option must be eligible--accept replaces the baseline--allow-network for real endpoints--min-accuracy exits 2
Never doesDecide for the userRun JavaScript or fetch linked resourcesLoad testingDownload or start a model

Run the reference#

CommandWhat it does
node --experimental-strip-types cli.ts fixtures/exhibition.json /tmp/decision-boardWrites the decision board
python3 -m http.server 8123 --directory /tmp/decision-boardServes it on localhost:8123. Stop it after use
go run . --before fixtures/before.html --after fixtures/after.html --url <url> --out ./web-change-outputCompares two saved snapshots
go run . --fetch <url> --baseline ./web-change-output/baseline.json --out ./next-reportFetches one live page against the baseline
node cli.ts sample.json outputRuns the tutorial on a loopback fixture
python3 cli.py samples/input.json --output scorecard.json --html reliability.htmlScores the sample recording
python3 scripts/project_test.py <project> --all --solution --strictGrades the reference

Decision scoring rules#

Benefit
(v - c.min) / (c.max - c.min).
Cost
(c.max - v) / (c.max - c.min).
Score
sum of normalized value times weight, divided by the total weight.
Missing value
the option is ineligible, also when that weight is 0.
Limit
a hard limit. A cost above it or a benefit below it fails.
Ties
ordered by option id.
Dominated
another option is as good on all criteria and better on one.
Sensitivity
sweeps one weight from 0 to 1 in steps.

Web change brief#

HTMLblockstwo snapshotsbriefafterclientbefore.html, after.htmlor --fetch one URLprocessExtractBlocks()readable text blocksprocessNewSnapshot()canonical URL, hashprocessCompare()added, removed, unchangedstoreExportBrief()index.html, changes.jsonstoreSaveSnapshot()baseline.jsonHTMLblockstwo snapshotsbriefafterclientbefore.html,after.htmlor --fetch one URLprocessExtractBlocks()readable textblocksprocessNewSnapshot()canonical URL,hashprocessCompare()added, removed,unchangedstoreExportBrief()index.html,changes.jsonstoreSaveSnapshot()baseline.json

A tutorial request block#

The tutorial holds JSON inside a request fence
{"id": "create-loan", "method": "POST", "path": "/loans",
"status": 201,expected status, 100..599
"capture": {"loanId": {"path": "id", "type": "number"}}}typed capture
{"id": "read-loan", "method": "GET",
"path": "/loans/{{loanId}}",only declared variables
"status": 200,
"assert": {"bookName": "Neighborhood Trees"}}stale field, fails

API tutorial runner steps#

1extractRequestsreads each request fence. It allows 20 steps atmost.2CLIstarts a loopback server from fixture when thereis no endpoint.3runTutorialfills {{name}} from earlier captures. Anundeclared name fails.4runTutorialsends the request with no redirects, a timeoutand a byte limit.5runTutorialchecks the status and each assert path, thenstores captures.6runTutorialafter one failure, it skips the next steps andsends nothing.7exportResultswrites JUnit, HTML and corrections.json for eachfailure.

Record and compare a local model#

python3 record.py samples/cases.json \loopback HTTP endpoint only
--endpoint http://127.0.0.1:11434/v1/chat/completions \
--model <model> --revision <rev> --hardware <hw> \
--output measured.jsonmanifest, source, labels, records
python3 cli.py measured.json --baseline samples/input.json \
--output compared.jsonrejects different dataset labels
python3 cli.py <recording> --min-accuracy <n>exit 2 below the gate

Scorecard fields#

accuracy
exact answer after case folding and space cleanup, over all labels.
coverage
predictions divided by labels.
ece
confidence gap per bin, weighted by bin count.
p50_ms, p95_ms
nearest rank latency percentiles.
dataset_sha256
hash of the labels. A baseline must match it.
manifest
needs model, model_revision, prompt_revision, hardware and confidence_method.

Defaults and limits#

SettingDefaultNote
--outweb-change-outputWeb brief report folder
MaxHTMLBytes2_000_0002 MB per page
timeoutMs1500Per request, 1 to 30000
maxBytes65536Response byte limit
Request steps20At most per tutorial
bins5Calibration bins
--min-accuracy0Exit code 2 below it
Sensitivity steps101 to 100

Error messages and fixes#

SymptomCauseFix
chosen option must be eligibleOption has a missing value or breaks a limitChoose a scored option
value outside declared rangeValue is not within min and maxFix the value or the range
Snapshots must refer to the same canonical URLTwo different URLsCompare snapshots of one page
Live fetch requires --baseline JSON--fetch without a baselineRun the saved-file mode first
Baseline checksum mismatchBaseline file was editedMake a new baseline
Network endpoint requires explicit allowNetworkEndpoint is not loopbackAdd --allow-network
Earlier step failed; no request sentA previous step failedFix the first failed step
cannot compare different datasetsLabels differ from the baselineRecord the same cases again

Worked examples#

Which option wins the exhibition sample?
pods with 0.68. open has 0.67.
Why is archive not ranked?
Its flow value is null, so it is ineligible.
When does open win?
In the access sweep, at weight 0.6 or more.
Why does read-loan fail?
It asserts the stale field bookName.
What does the eval sample score?
2/3 accuracy, full coverage, p95 of 200 ms.

Tips#

  • Use one filter policy for both snapshots. Filter phrases can hide changes.
  • The baseline stays untouched unless you pass --accept.
  • Weights are preferences, not probabilities.
  • Reported confidence is model self-report. ECE changes with bins and sample count.
  • The tutorial command succeeds after it writes evidence. Read the receipts.