SRE, Incidents and Postmortems for AI

Triage LLM incidents with agents, gate every fix, write evidence-bound postmortems and compact logs for an assistant.

Incident symptoms#

Symptom	Cause	Fix
High error rate in checkout	vLLM OOM from KV cache spike	Restart pod. Lower `--gpu-memory-utilization` to 0.85
GPU memory at 98% before the spike	KV cache OOM under burst concurrency	Apply runbook RB-017
Two models disagree on root cause	Possible hallucinated root cause	Escalate with both hypotheses visible
Pod OOMKilled twice in 10m	Memory request 256Mi too low	Request 512Mi, limit 1Gi
CrashLoopBackOff on auth-svc	Probe path changed to /ready	Revert the probe path to /healthz
Image `checkout-api:v2.41` fails /healthz	Bad rollout 14m ago	Roll back to v2.40 after approval
`blocked: no slack approval`	Destructive tool with no approver	Get the Slack card approved
Rare disk-full line lost from context	Uniform sampling with budget 8	Use the `rarity` policy

Factor	Safe, narrow	Not safe, broad
Pods and code	Restart pod	Deploy new code
Deploys	Revert a specific deploy	Change service topology
Capacity	Scale within pre-approved bounds	Modify resource limits
Config and access	Enable a pre-approved feature flag	Change IAM, alter databases
RBAC verbs	get, list, describe	delete, exec, scale
Capstone tools	`kubectl_get`, `promql`, `logql`	`kubectl_delete`, `argocd_rollback`
Gate	Runs at once	Needs Slack approval

def score(self) -> float:

recency_w = max(0.0, 1.0 - self.recency_mins / 60.0)0 after 60 min

path_w = 1.0 / (1 + self.path_len)short graph path wins

return (recency_w * 0.35 +

self.specificity * 0.35 +0..1

min(len(self.citations), 5) / 5 * 0.2 +caps at 5 citations

path_w * 0.1)

Task	Code
Multi-agent triage simulator, lesson 23	`code/main.py`
K8s graph agent with approval gate	`python main.py`
Slack skeleton self-check, then tests	`npm start` and `npm test`
Slack interactive server	`npm run serve`
Postmortem packet from the examples	`go run . --events ../examples/events.jsonl --review ../examples/review.json --out /tmp/incident-packet`
Build the log context tool	`rustc --edition 2021 main.rs -o /tmp/log-context`
Compact logs, budget 8, rarity	`/tmp/log-context fixtures/burst.log 8 rarity /tmp/context.jsonl`
Rare-template coverage report	`python3 main.py fixtures/burst.log /tmp/context.jsonl --out /tmp/coverage.json`

Setting	Default	Note
`REPLAY_WINDOW_SECONDS`	60 * 5	Older Slack requests are stale
`SIGNATURE_VERSION`	v0	HMAC sha256 over v0:ts:body
`rare_max`	2	Rare template frequency
Log input size	1_048_576	Bytes, one MiB
Log records	10_000	Lines per input
Log line length	4096	Bytes per line
Event JSONL size	2<<20	Postmortem events
Claims and actions	100	Each, per review

Command	What it does
`review source changed`	Review hash differs from the events file
`approval requires matching source and reviewer`	Approved claim with no hash or reviewer
`quote for <id>`	Quote is blank or not in the message
`budget exceeded`	Too many events, claims or bytes
`conflict`	Duplicate event ID or evidence ID
`expected LEVEL<TAB>message`	Log line has no tab
`group count mismatch`	Context count differs from the source
`invalid locator`	firstLine or lastLine is out of range

What does the audit log record?: Every command considered, executed or approved, and who approved it.
What is the capstone bar?: At least 80% correct root cause on 20 synthetic incidents.
How fast is a hypothesis?: p50 under 5 minutes from alert to Slack brief.
Can prediction alone help?: No. Predictions without actuation are dashboards.
Does the packet prove the cause?: No. Provenance is checked. Causality requires review.
Uniform or rarity sampling?: Rarity. Uniform with budget 8 misses line 138.

Write runbooks with symptom, hypothesis, verify and act sections.
Store runbooks and post-mortems in a vector DB for retrieval.
Before chaos, have SLI/SLO, observability, automated rollback, runbooks and on-call.
Hold a monthly game day. End it with a postmortem.
Pre-redact logs before you send context to an assistant.