Cheat sheets

Production Agent Runtimes and Queues

Pick a runtime shape, checkpoint and replay, lease and fence jobs, retry scoped reads. Python and Go.

  • Concepts and terms
  • Tasks and commands
  • Annotated code
  • Diagrams
  • Defaults and numbers
  • Troubleshooting
  • Questions and tips

Source

Pick the runtime shape#

Request-response
synchronous HTTP. The user waits. Short tasks only (<30s).
Streaming
SSE or WebSocket. Watch first-token latency and tail latency.
Durable execution
state checkpointed after every step. Auto-resumes on failure.
Queue-based
producer, worker pool, DLQ. Watch queue depth and DLQ size.
Event-driven
the agent subscribes to triggers. Watch event-to-start latency.
Scheduled
cron-shaped. Combine with durable execution.
Workflow and activity
deterministic orchestration code. Each LLM or tool call is a logged activity.
Exactly-once effective
at-least-once delivery plus an idempotent consumer.

From Lesson 14.29, Lesson 15.12, Lesson 16.22

Durable job store#

read-modify-writesnapshottextClaimNextreceiptclientenqueue--store, --inputstoreledger.lockadvisory lockstorejobs.jsonID, State, Versionprocessworkerclaim, effect, completestoreinputs/<id>.jsonimmutable inputstoreeffects/<id>.jsonimmutable receiptread-modify-writesnapshottextClaimNextreceiptclientenqueue--store, --inputstoreledger.lockadvisory lockstorejobs.jsonID, State, Versionprocessworkerclaim, effect, completestoreinputs/<id>.jsonimmutable inputstoreeffects/<id>.jsonimmutable receipt

From durable-agent-jobs/API.md, durable-agent-jobs/README.md

Job lifecycle in the stages#

1NewJobthe ID must match ^[a-z][a-z0-9-]{0,63}$. Stateis queued, version 0.2ClaimJobneeds the expected version and state queued.Else ErrConflict.3ClaimJobAttempts >= maxAttempts gives ErrLimit.4ClaimJobVersion++, Attempts++, LeaseUntil = now + ttl,state running.5Finishcurrent version, state running and now <LeaseUntil. State completed.6Reclaimrunning and now >= LeaseUntil. State queued,Version++ fences the old worker.7SaveValidateJobs, write temp, f.Sync(), os.Rename.Limit 4 MiB.

From solution/stage1.go, solution/stage2.go, solution/stage3.go, solution/stage4.go

Durable jobs CLI#

CommandWhat it does
go build -o /tmp/durable-jobs .Builds the reference binary in solution
enqueue --store <dir> --input <file>Adds jobs. Identical redelivery is safe
worker --store <dir> --max-jobs 100Claims and runs up to 100 jobs
status --store <dir>Shows jobs, available effects and scope
--crash-after effectExits 86 after the receipt exists
--crash-after claimExits 86 before an output exists
--now-ms 111 --lease-ms 10Logical clock. The old lease expired
go run . demoRuns the effect-crash sequence

From durable-agent-jobs/README.md, durable-agent-jobs/API.md

Run the demos#

TaskCode
Run a lesson demopython3 code/main.py
Grade a project reference--all --solution --strict
Run the cloud readerpython3 cli.py samples/input.json
Read live AWS insteadpython3 cli.py <input> --mode aws
Pin the optional SDKstrands-agents==1.57.1

From Lesson 14.29, durable-agent-jobs/README.md, cloud-agent-with-aws-strands/README.md, cloud-agent-with-aws-strands/requirements-framework.txt

Activity replay from an event log#

@activity("call_llm")each LLM call is an activity
def call_llm(doc_count: int) -> str:
return f"summary({doc_count}_docs)"
hit = log.lookup(name, args)done event with the same args
if hit:
return hit["result"]replay, no new execution
log.append({"name": name, "args": list(args), "status": "started"})
result = fn(*args)runs only when not in the log
log.append({"name": name, "args": list(args),
"status": "done", "result": result})

From 15.12 code/main.py

Checkpoint per super-step#

store = CheckpointStore(db_path)SQLite table checkpoints
store.write(thread_id, super_step, dict(state))one row per step
restored = store.latest(thread_id)ORDER BY super_step DESC
super_step, state = restored
super_step += 1resume after the last step
run_agent_with_checkpoint(store, "t-1", crash_at=3, goal=5)
final = run_agent_with_checkpoint(store, "t-1", goal=5)worker 2
await asyncio.gather(*(sim_llm_call_async() for _ in range(n)))

From 16.22 code/main.py

Scoped cloud reader contract#

ALLOWED = {"inventory.list", "metrics.read", "logs.read"}
validate_plan(raw, scope)one to ten actions, read-only, in scope
execute(plan, provider, max_steps=5, max_chars=4000)
over budget: state budget_exhausted
request_key(operation, resource) # sha256 of the JSON pair
cached_read(operation, resource, provider, cache, retries=2)
cache hit: attempts 0. Only TimeoutError retries
run(payload, provider)adds receipts with cached, attempts
parse_model_plan(text, scope)the model plan goes to validate_plan

From cloud-agent-with-aws-strands/API.md, solution/plan.py, solution/executor.py, solution/retry.py, solution/cli.py

Defaults#

SettingDefaultNote
--lease-ms30000Claim lease in milliseconds
--max-attempts3Then the job waits for manual reconciliation
--max-jobs1Jobs for one worker invocation
--delay-ms0Range 0..60000
--now-ms-1-1 means current wall time
max_steps5Step budget. max_chars is 4000
retries2In cached_read. The CLI run passes 1

From durable-agent-jobs/API.md, solution/main.go, solution/executor.py, solution/retry.py, solution/cli.py

Troubleshooting#

SymptomCauseFix
Users hang up, retries compoundRequest-response for a 5-minute taskUse a queue or durable shape
Failed jobs vanishQueue workers without dead-letterAdd a DLQ and surface DLQ size
Failures invisible until a user reportsBackground run without trace exportExport OpenTelemetry GenAI spans
Crash reruns tools and re-bills LLM callsNaive while True loopLog each call as an activity and replay
Completion returns conflictStale worker: version or lease changedThe new claim finishes the job
Job stays at the attempt cap--max-attempts reachedReconcile the visible job manually
Replay divergesWall clock inside a workflow decisionRegister it as a side effect
10,000 calls use 10GB of stacksOne thread for each LLM callUse async for the I/O layer

From Lesson 14.29, Lesson 15.12, Lesson 16.22, durable-agent-jobs/README.md, durable-agent-jobs/API.md, solution/stage3.go

Questions#

Which checkpoint backend?
PostgreSQL survives deploys. SQLite is local-dev only. Redis is ephemeral unless AOF/snapshot configured.
When is durable execution wrong?
Runs shorter than a few minutes with no human input, or strictly read-only retrieval.
Does durability fix the 35-minute degradation?
No. Add fresh HITL on re-entry and budget kill switches.
Is the receipt exactly once?
No. Execution is at least once. The receipt makes only this local effect idempotent.
Why two provider reads for three steps?
Repeated metrics.read uses the request-local cache.
What does recovery show?
incident-summary is completed v4, attempts 2, with reused_effect: true.

From Lesson 15.12, durable-agent-jobs/README.md, cloud-agent-with-aws-strands/README.md, Lesson 16.22

Tips#

  • Start with FastAPI + Postgres. Upgrade when you measure it failing.
  • Put a dedup key in every side-effect call.
  • Use the outbox pattern for payments and external API calls.
  • Use rainbow deploys. Never kill in-flight agent runs during deploys.
  • Use async for I/O. Use threads only for CPU-bound post-processing.
  • Keep the job store on one local POSIX filesystem. Do not put it on NFS.
  • Do not edit storage files while workers run.

From Lesson 16.22, durable-agent-jobs/README.md, durable-agent-jobs/API.md