Model Routing as a Cost-Reduction Primitive
A dynamic broker evaluates every request (task type, token length, embedding similarity, confidence) and sends simple queries to a cheap model, escalating complex ones to a frontier model. Also called model cascading. Production case studies show 20-60% cost reduction at iso-quality across US/UK/EU deployments; a 30% routing efficiency improvement on high-volume SaaS turns into six-figure annual savings. The 2026 context is that LLM inference prices dropped 10x per year — a GPT-4-class token went from $20/M to $0.40/M from late 2022 to 2026. Most of the drop is better serving stacks (Phase 17 · 04-09), not hardware. Routing is how you convert that price drop into margin without product regression. The failure mode is cheap-model drift: the route pushes 40% to a weaker model, quality drops 3-5% on reasoning tasks, no one notices for a quarter. Gate routes by online quality metrics, not just offline eval sets. Explain model cascading: cheap-first with confidence check, escalate on low confidence. Enumerate the four routing signals (task classification, prompt length, embedding similarity to known-hard set, self-confidence from first-pass). Compute expected blended cost at target routing split and quality loss tolerance. Name the drift-monitoring metric (online quality gate) that catches cheap-model creep. Your service costs $80k/month on GPT-5. Your analytics show 70% of queries are simple: "what time is it in Paris?" "rephrase this…
Model Routing as a Cost-Reduction Primitive: A dynamic broker evaluates every request (task type, token length, embedding similarity, confidence) and sends…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.