Shadow, Canary and A/B Tests for LLMs

Shadow runs, canary gates, fast rollback, sequential A/B tests, chaos guards and config drift reports.

Canary gates in the simulator#

Setting	Default	Note
`STAGES`	0.01, 0.10, 0.25, 0.50, 0.75, 1.00	Six traffic shares
`latency_p99_ms`	1.5	Baseline 900 ms
`cost_per_req`	1.2	Baseline 0.02. Breach above 20%
`error_rate`	2.0	5xx plus explicit refusals
`output_len_p99`	1.4	Baseline 450 tokens
`thumbs_down_rate`	1.5	Baseline 0.03
`rng.uniform` noise	0.92, 1.08	Noise on each measured metric

required = fixed_sample_size(baseline, lift)80% power, alpha 0.05

adjusted = int(required * 1.4)LLM non-determinism buffer

if n_a > 100 and n_b > 100 and sequential_stopped_at is None:

z = z_statistic(success_a, n_a, success_b, n_b)

n_total = n_a + n_b

threshold = math.sqrt(2 * math.log(1 / 0.05) + math.log(n_total))grows with log(n)

if abs(z) > threshold:

sequential_stopped_at = n_totalstop early on a clear winner

Command	What it does
`go run . --input fixtures/config.json --out /tmp/drift.json`	Write the drift report to a file
`go build -o /tmp/drift-explainer .`	Build the binary
`/tmp/drift-explainer --input fixtures/config.json --check`	Return 2 when drift exists
`python3 scripts/project_test.py configuration-drift-explainer --init <work>`	Make a fresh starter
`python3 scripts/project_test.py configuration-drift-explainer --stage 1 --path <work> --strict`	Grade one stage
`python3 scripts/project_test.py configuration-drift-explainer --all --solution --strict`	Grade the reference solution

Symptom	Cause	Fix
Cost up 40% a day after the flip	No shadow or canary step	Shadow first. Gate cost at 20% over baseline
Rollback takes 3 hours	Rollback needs a redeploy	Flip the policy flag. Pin the old digest
Canary gates false alarm	Gates sit inside the noise floor	Set gates above the noise floor
Feedback gate noisy at 1%	Only 50-150 data points per window	Pause longer at each step
50/50 split gives 47/53	SRM. Assignment is broken	Fix the assignment hash first
A/B never reaches significance	Non-determinism lowers effective sample size	Multiply sample size by 1.3 to 1.5
OOM after provider 429s	Gateway retries amplify concurrency	Inject one provider 429 for 5 minutes
Secret value in the drift report	Innocent key name hides a secret	Add the key to `sensitive`

Task	Code
Canary simulator, lesson 20	`code/main.py`
Inject a 25% cost regression	`Regression(cost_mult=1.25)`
Inject an 80% latency regression	`Regression(latency_mult=1.80)`
Check one stage against the gates	`check_gates(metrics)`
Sequential A/B simulator, lesson 21	`code/main.py`
Sample size at 80% power	`fixed_sample_size(baseline, lift)`
Simulate a 5% lift	`simulate(0.03, 0.0315)`
Chaos runner, lesson 24	`run_experiment(e)`

Shadow mode: zero user impact. A smoke test, not a quality test.
Canary: gradual rollout to real users, with metric gates.
Non-determinism: up to 15% accuracy variation run to run on identical inputs.
Policy flag: rollback in config. It takes seconds, not hours.
Model pin: immutable registry digest. It does not auto-upgrade.
Eval vs A/B: an eval asks "can the model do the job?" An A/B test asks "do users care?"
CUPED: regress out pre-period variance. Typical reduction is 30-70%.
SRM: sample ratio mismatch. A 50/50 split that gives 47/53 is broken.

Command	What it does
`ERROR_BUDGET_PER_DAY = 0.001`	99.9% SLO
`EXPECTED_ERROR_RATE = 0.0005`	Baseline error rate
`burn_rate > 2.0`	First half of the abort rule
`e.blast_radius_pct > 0.2`	Second half. Both must be true
`pod kill (1 decode replica)`	5 min, 5% blast. Completes
`provider 429 fallback`	5 min, 30% blast. Aborts
`malformed prompt tokenizer stall`	3 min, 10% blast. Completes

Input: layers, observed and optional sensitive. Later layers win.
equal: the effective value and the observed value match.
changed: the values differ under reflect.DeepEqual.
missing: the layers declare the key. The observed config does not.
unexpected: the observed config has a key that no layer declares.
History: every layer that set the key, in order. Shadowed values stay.
Redaction: password, secret, token, api-key and credential names become [REDACTED].
Output: schemaVersion 1. The file has owner-only permissions 0600.

Does shadow mode prove quality?: No. It catches cost, length, refusal and hard errors only.
When do you skip the A/B step?: When the new model is only an improved version.
Is "stable" equal to baseline?: No. Stable means within expected variance.
20 tests at 95% confidence?: One false positive by chance. Use Bonferroni or Benjamini-Hochberg.
Which chaos experiment aborts?: Provider 429. Burn rate is over 2.0 and blast radius is over 20%.
What does --check return on drift?: Exit status 2.