Prompt Caching and Semantic Caching Economics
Pricing snapshot dated 2026-04. Numeric claims below reflect vendor rate cards captured at this lesson's publication; verify against the linked docs before quoting them downstream. Distinguish L2 prompt/prefix caching (KV reuse at provider) from L1 semantic caching (LLM bypass on similar prompts). Explain Anthropic's cachecontrol explicit marking and the two TTL options (5-min vs 1-hour) with their price multipliers. Compute expected monthly savings given hit rate, prompt/response mix, and token prices. Name the parallelization anti-pattern that inflates bills by 5-10x and the dynamic-content anti-pattern that collapses hit rate. You add prompt caching to your RAG service. The bill stays flat. You measure the hit rate; it is 7%. Your prompts look static but they are not — the system prompt includes the current date formatted to the minute, a request ID, and a randomized example reorder for diversity. Every request writes a new cache entry, reads zero. Separately, your agent runs ten parallel tool calls per user question. All ten arrive at the provider before the first cache write completes. Ten writes, zero reads. Your bill is 5-10x what "with caching" was supposed to cost. Caching is a protocol, not a flag. Two layers, two different failure modes. Provider stores the attention KV for a cacheable prefix and reuses it on the next request that matches the prefix. You pay…
Prompt Caching and Semantic Caching Economics: Pricing snapshot dated 2026-04. Numeric claims below reflect vendor rate cards captured at this lesson's…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.