AI-native development · Glossary term
What is Model Router?
A component that selects a model or provider for a request using requirements such as capability, latency, cost, context size, policy, and current availability.
Why does Model Router matter?
Different tasks and failure conditions justify different models, and routing can improve outcome quality without sending every request to the largest option.
Model Router in practice
Send low-risk extraction to a fast model, complex code review to a stronger model, and fail over only to providers that satisfy the same data policy.
What is the common confusion about Model Router?
Routing is a policy decision. Random load balancing only distributes traffic.
Learn Model Router in the course
Lessons that name Model Router in a title or section
- Caching, Rate Limiting & Cost Optimization
Most AI startups do not die from bad models. They die from bad unit economics. A single GPT-4o call costs fractions of a cent.
Covered in Phase 11: LLM Engineering.
Related terms
- Evaluation (Eval)A defined process for measuring model or system behavior on representative tasks using explicit success criteria, data, scorers, and…
- Circuit BreakerA reliability control that temporarily stops calls to a dependency after failures cross a threshold, then probes whether the dependency…
- Rate LimitA policy that caps requests, tokens, concurrent work, or another resource within a defined time or capacity window.
- Cost per Successful TaskTotal system cost divided by the number of tasks that satisfy a defined success criterion, including retries, failed runs, tool use, and…
- Graceful DegradationPreserving a bounded core service when capacity or dependencies are impaired by reducing optional quality, features, freshness, or…
- Model ServingThe runtime and API layer that loads versioned model artifacts, accepts inference requests, schedules execution, manages resources, and…
- MoE (Mixture of Experts)An architecture with multiple expert subnetworks and a learned router that selects a subset for each input unit, often each token.
More terms in AI-native development
This entry comes from glossary/terms.md on GitHub. Browse all 250 glossary terms.