Caching, Rate Limiting & Cost Optimization
Most AI startups do not die from bad models. They die from bad unit economics. A single GPT-4o call costs fractions of a cent. Ten thousand users making ten calls per day costs $250 in input tokens alone -- before you charge a single dollar. The companies that survive are the ones that treat every API call as a financial transaction, not a function call. Related: Phase 11 · 15 (Prompt Caching) — this lesson covers application-layer caching (semantic cache, exact hash cache, model routing). Lesson 15 covers provider-layer prompt caching (Anthropic cachecontrol, OpenAI automatic, Gemini CachedContent). Combine both for 50-95% cost reduction. Implement semantic caching that serves repeated or similar queries from cache instead of making a new API call. Calculate per-request costs across providers and implement token-aware rate limiting and budget alerts. Build a cost optimization layer with prompt compression, model routing (expensive vs cheap), and response caching. Design a tiered caching strategy using exact match, semantic similarity, and prefix caching for different query types. You build a RAG chatbot. It works beautifully. Users love it. Then the invoice arrives. GPT-5 costs $5 per million input tokens and $15 per million output. Claude Opus 4.7 costs $15 input / $75 output. Gemini 3 Pro costs $1.25 input / $5 output. GPT-5-mini is $0.25/$2. Prices below are illustrative; always…
Caching, Rate Limiting & Cost Optimization: Most AI startups do not die from bad models. They die from bad unit economics. A single GPT-4o call costs…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.