Shipping a Production LLM App

Request pipeline, retries, fallback chains, SSE streams, resume, demo limits and a fallback gateway.

Tasks and commands
Annotated code
Diagrams
Comparisons
Defaults and numbers
Troubleshooting
Questions and tips

Lesson tasks#

Task	Code
Install the server stack	`pip install fastapi uvicorn`
Run the app with 4 workers	`uvicorn production_app:app --host 0.0.0.0 --port 8000 --workers 4`
Pick the primary model	`LLM_MODEL=gpt-4o`
Start the demo server	`python3 code/main.py --serve`
Require an access token	`export DEMO_ACCESS_TOKEN=<token>`
Send one prediction	`curl -s -X POST http://127.0.0.1:8000/api/predict -H 'Content-Type: application/json' -d '{"text": "ok"}'`
Watch the event stream	`curl -N -X POST http://127.0.0.1:8000/api/stream -H 'Content-Type: application/json' -d '{"text": "<text>"}'`
Run the demo tests	`python3 -m unittest discover -s code/tests -v`

From Lesson 11.13, 11.13 code/production_app.py, Lesson 17.29, 17.29 code/main.py

Request pipeline#

From Lesson 11.13, 11.13 code/production_app.py

Retry with backoff#

async def call_llm_with_retry(prompt, model, max_retries=3):

for attempt in range(max_retries + 1):

try:

...one provider call

except (ConnectionError, TimeoutError):

if attempt < max_retries:

backoff = min(2 ** attempt + random.uniform(0, 1), 10)cap 10 s

await asyncio.sleep(backoff)

else:

raise

raise ConnectionError(f"All {max_retries} retries exhausted for {model.value}")

call_with_fallback walks FALLBACK_CHAIN, preferred model first.

From 11.13 code/production_app.py

Failure handling#

Symptom	Cause	Fix
API 429	Rate limit	Retry with backoff. Queue the request
API 500	Server error	Retry 3 times. Switch to the fallback model
API timeout over 30s	Slow provider	Retry once. Shorter prompt, smaller model
Malformed output	Model failure	Retry with the error in the prompt
Guardrail block	Unsafe input	No retry. Explain the block
Vector store down	Application failure	Skip RAG context
Cache down	Application failure	Direct LLM call
All models fail	Chain exhausted	Cached response, then a fixed apology

From Lesson 11.13

Defaults in the code#

Setting	Default	Note
`similarity_threshold`	0.92	Semantic cache hit score
`ttl_seconds`	3600	Cache entry life
`max_entries`	10000	Cache size
`max_retries`	3	Per model in the chain
`MAX_CHARS`	500	Demo text limit
`MAX_BODY_BYTES`	4096	Larger body gives 413
`BUCKET_CAPACITY`	5	Refill 0.5 token per second
`DAILY_CALL_CAP`	2000	Over the cap gives 503

From 11.13 code/production_app.py, 17.29 code/main.py

Demo error codes#

Command	What it does
`413 body_too_large`	Body over 4,096 bytes
`408 request_timeout`	Body stops before `Content-Length`
`415 unsupported_media_type`	Content type is not JSON
`400 bad_json`	Body is not a JSON object
`422 text_too_long`	Text over 500 characters
`422 control_characters`	Text has control characters
`429 rate_limited`	Token bucket empty. Read `Retry-After`
`503 usage_cap_reached`	Daily cap reached. Read `Retry-After`

From Lesson 17.29, 17.29 code/main.py

SSE wire format#

event: tokenContent-Type text/event-stream

data: {"index":0,"text":"Label:"}

a blank line ends each event

event: token

data: {"index":1,"text":" positive."}

event: done

data: {"label":"positive","score":1.0,"tokens":11}

From Lesson 17.29

Resume a streamed answer#

From streaming-answer-recovery/README.md, solution/main.ts, solution/cli.ts

Gateway fallback policy#

Factor	2xx	429 or 5xx	Other status	Cancellation
`ClassifyStatus`	`success`	`retry`	`terminal`	none
Next provider	No	Yes	No	No
Route state	`completed`	`exhausted` at end	`terminal-error`	`cancelled`
Proxy reply	Upstream body	502 at end	502	504

From llm-gateway-with-fallbacks/README.md, solution/stage2.go, solution/stage4.go, solution/config.go

Gateway providers file#

{"providers": [

{"name": "primary", "url": "http://127.0.0.1:11434/v1/chat/completions"},

{"name": "backup", "url": "http://127.0.0.1:11435/v1/chat/completions",

"key_env": "ORCHARD_BACKUP_KEY"}],env var name, not the key

"max_attempts": 2,1 to 16, one visit per provider

"max_response_bytes": 65536,1 byte to 16 MiB

"timeout_ms": 4000}1 to 5000, whole route

From fixtures/providers.example.json, llm-gateway-with-fallbacks/README.md

Project tasks#

Task	Code
Make a work folder	`python3 scripts/project_test.py <project> --init /tmp/<project>-work`
Grade one stage	`python3 scripts/project_test.py <project> --stage 1 --path /tmp/<project>-work --strict`
Stream demo, drop then resume	`cd projects/streaming-answer-recovery/solution && python3 demo.py`
Read your own SSE endpoint	`node --experimental-strip-types cli.ts <url> /tmp/trace.json`
Gateway demo, 503 then backup	`cd projects/llm-gateway-with-fallbacks/solution && go run .`
Run the gateway proxy	`go run . --config /tmp/providers.json --listen 127.0.0.1:8088`
Call the proxy	`curl --fail-with-body http://127.0.0.1:8088/v1/chat/completions --data-binary @fixtures/request.json`

From streaming-answer-recovery/README.md, llm-gateway-with-fallbacks/README.md

Targets and checklist#

P50 latency under 2s. P99 latency under 10s.
Cache hit rate over 30%. Guardrail block rate under 5%.
Cost per request under $0.01.
Rate limit each user at 10 to 50 requests per minute.
Load test with 100 concurrent users before launch.

From Lesson 11.13

Q&A#

Why POST and not EventSource?: EventSource sends GET only. GET puts the input in logged URLs.
Model fails after status 200?: The status is sent. Send an error event in the stream.
What is in an error body?: code, message, hint and request_id.

From Lesson 17.29