LLM Routing Layer — LiteLLM, OpenRouter, Portkey
Provider lock-in is expensive. Different tool-calling workloads suit different models. Routing gateways give one API surface, retries, failover, cost tracking, and guardrails. Three archetypes dominate 2026: LiteLLM (open-source self-hosted), OpenRouter (managed SaaS), Portkey (production-grade, open-sourced in March 2026). This lesson names the decision criteria and walks a stdlib routing gateway. Distinguish self-hosted, managed, and production-grade routing options. Implement a fallback chain that retries on provider failures in a defined priority order. Track per-request cost and token usage across providers. Decide between LiteLLM, OpenRouter, and Portkey for a given production constraint. Scenarios where provider routing matters: Cost. Claude Sonnet costs 3x what Haiku costs. For a triage task, Haiku is enough; for a synthesis task, Sonnet is worth it. Route per-request. Failover. OpenAI has a bad hour. Every request fails. You want automatic fallback to Anthropic without redeploying. Latency. A live chat UI needs fast time-to-first-token. A batch summarizer does not. Route by latency SLA. Compliance. EU users must stay in EU regions. Route by region. Experimentation. A/B two models on the same workload. Route by test bucket. Hand-coding all of this per integration is repetitive. A routing gateway gives one OpenAI-compatible API and handles the rest. Everyone speaks OpenAI-shape. The routing gateway exposes /v1/chat/completions, accepts the OpenAI schema, and internally proxies to Anthropic / Gemini / Cohere / Ollama /…
LLM Routing Layer — LiteLLM, OpenRouter, Portkey: Provider lock-in is expensive. Different tool-calling workloads suit different models. Routing gateways give…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.