Cheat sheets

LLM Routing and AI Gateways Compared

Pick a gateway, set aliases and fallback chains, route cheap or frontier, and hedge slow safe reads.

  • Concepts and terms
  • Tasks and commands
  • Annotated code
  • Diagrams
  • Comparisons
  • Defaults and numbers
  • Troubleshooting
  • Questions and tips

Source

AI gateways side by side#

FactorLiteLLMPortkeyKong AI GatewayCloudflare / Vercel
CodeMIT OSS, PythonApache 2.0 OSS since March 2026Built on Kong GatewayManaged, zero-ops
Overhead5-15 ms20-40 ms3-8 ms1-3 ms at the edge
ScaleBreaks at ~2000 RPS, 8 GBControl plane228% faster than PortkeyEdge advantage
Rate limitToken-bucketTieredSliding-windowLimited
GuardrailsOptionalOpinionated guardrails, PII redactionOpinionated guardrailsLimited
Price$0 for OSS$49/mo production tier$100/model/month, max 5 on PlusManaged only
Best fitPython, <500 RPS, devRegulated industriesAlready on Kong, >1000 RPSEdge JavaScript apps

Run the labs#

CommandWhat it does
python code/main.pyRouting gateway: normal, outage, PII and all-down scenarios
python3 scripts/project_test.py hedged-request-lab --init <dir>Creates a work folder. The starter fails until you implement it
python3 scripts/project_test.py hedged-request-lab --stage 1 --path <dir> --strictGrades one stage
python3 scripts/project_test.py hedged-request-lab --all --solution --strictGrades the reference solution
go run . --samples 3 --delay-ms 10 --out /tmp/hedge-report.jsonIn solution: baseline against hedged on loopback
go run . --primary <url> --secondary <url>Hedges your own two safe read endpoints
python3 scripts/project_test.py hedged-request-lab --all --path <dir> --strict --report <file>Writes the completion evidence

Defaults and limits#

SettingDefaultNote
hardSetThreshold0.88At or above it, the router pins frontier
maxAttempts2Demo retry budget per provider
Free tier bucketcapacity: 2refillPerSec: 0.5, third call gets 429
--delay-ms10Hedge delay, allowed 0 to 1s
--samples3Samples per policy, 1 to 100
--out/tmp/aiefs-hedge-report.jsonReceipt file
GET attempts2Body 64KiB, two-second ceiling, 100ms grace

Troubleshooting#

SymptomCauseFix
Cascading failures near 2000 RPSLiteLLM at its ceilingMigrate to Kong first
TTFT P99 over a 100 ms SLAPortkey adds 20-40 msUse Kong or Cloudflare
Escalation rate over 30%Cheap model is over-routedRoute less traffic to the cheap model
Quality drops with no alarmCheap-model driftGate routes by online quality metrics
Outage takes the alias downChain has one providerAdd a second failure domain
Fallback cascades raise costRetries without a budgetCount retries against a budget
A fast error wins the hedgeWinner not validatedRequire HTTP 200 and nonempty value
Cancel shows no saved workServer keeps workingRead fixtureWork counts

Use X when#

LiteLLM
you have an SRE team, want data sovereignty and stay below 500 RPS.
OpenRouter
managed SaaS, 300+ providers, credits. You want no infra.
Portkey
you need guardrails and compliance out of the box.
Kong AI Gateway
you already run Kong and need more than 1000 RPS.
Cloudflare / Vercel
you serve JavaScript at the edge and need zero ops.
Self-host
data residency forces it. Healthcare and finance default to self-host.
No gateway
a single-model prototype. Direct API calls are simpler.

Routing patterns#

FactorPre-routeCascadeEnsemble
HowClassifier up frontCheap first, escalateCheap and frontier in parallel
Latency~5-10ms added, fastest~1.2x median, ~2x escalatedParallel runs
QualityDepends on classifierBest quality floorHighest
CostCheap for simple onlyCheap plus frontier on escalationHighest cost
Use whenClassifier is accurateQuality floor mattersCritical A/B only

Gateway request path#

handletenantallowedeach providercallclientClientpresented API keyprocessAuthService401 invalid api keyprocessTokenBucketLimiter429 rate limit exceededprocesscallWithFallbackchain in orderprocesscallWithRetrybackoff on retryablemodelProvidersopenai, anthropic,self-hostedhandletenantallowedeach providercallclientClientpresented API keyprocessAuthService401 invalid api keyprocessTokenBucketLimiter429 rate limitexceededprocesscallWithFallbackchain in orderprocesscallWithRetrybackoff on retryablemodelProvidersopenai, anthropic,self-hosted

One request: route()#

1redact_piiSSN and 16-digit card patterns become[REDACTED].2routeROUTES.get(alias, [alias]) gives the fallbackchain.3provider_calla model in OUTAGE raises a simulated 5xx.4routeon RuntimeError it tries the next model in thechain.5Costprompt_tokens times in_rate pluscompletion_tokens times out_rate, over1_000_000.6routewhen every model fails, error is "all providersfailed".

Alias routing gateway in code#

ROUTES = {"smart": [...], "fast": [...]}alias to fallback chain
PRICES = {"openai/gpt-4o": (5.0, 15.0)}per 1M tokens, in and out
inv = route("smart", [{"role": "user", "content": "<prompt>"}])
inv.chosen_modelfirst model that answered
inv.attemptsevery model tried, in order
inv.input_tokens, inv.output_tokensfrom usage
inv.cost_usdtokens times PRICES
inv.redactedTrue when PII matched
inv.error"all providers failed"
OUTAGE.add("openai/gpt-4o")simulate a primary outage

Tasks in code#

TaskCode
Route an aliasroute(alias, messages)
Scrub PII before dispatchredact_pii(text)
Compare routing patternssimulate("CASCADE", reqs)
Pick a model by policyrouter.pick(c.signals, c.policy)
Gateway fallback simulatorsimulate_fallback(gateway, n=1000, seed=7)
Retry one providercallWithRetry(provider, prompt, cfg)
Try providers in ordercallWithFallback(chain, prompt, cfg)
Hedge a safe readHedged(ctx, client, endpoints, delay)

Numbers and questions#

What does fallback buy?
One provider at 3% error gives 97%. Two give 99.94%. Three give 99.997%.
What does routing save?
20-60% at iso-quality in case studies.
What does a 70/30 split do?
The bill drops ~65% at the same product quality.
What does semantic caching save?
30 to 60 percent on repeated agent loops.
Which online gates catch drift?
Thumbs per route, LLM-judge on a 5% sample, escalation and refusal rate.
Which metric to alert on first?
Fallback-use rate, then gateway latency P99.