LLM Cost, Caching and Batch APIs

Price each call, cache prefixes and meanings, batch offline work and cap spend. Python and Rust from the lessons.

Concepts and terms
Tasks and commands
Annotated code
Diagrams
Comparisons
Defaults and numbers
Troubleshooting
Questions and tips

Tasks#

Task	Code
Price one call	`calculate_cost(model, input_tokens, output_tokens, cached_input_tokens=0)`
Mark a cacheable block	`"cache_control": {"type": "ephemeral"}`
Keep the block for one hour	`"cache_control": {"type": "ephemeral", "ttl": "1h"}`
Read Anthropic cache hits	`response.usage.cache_read_input_tokens`
Read OpenAI cache hits	`resp.usage.prompt_tokens_details.cached_tokens`
Create a Gemini cache	`client.caches.create(model=<model>, config=...)`
Use the Gemini cache	`config=types.GenerateContentConfig(cached_content=cache.name)`
Route a query to the cheapest model	`route_model(query, tier="pro")`

From Lesson 11.11, 11.11 code/caching_cost.py, Lesson 11.15

Cache the stable prefix#

SYSTEM = [{

"type": "text",

"text": RUBRIC,stable text first

"cache_control": {"type": "ephemeral"},5 min TTL

}]

response = client.messages.create(

model="<model>", max_tokens=1024,

system=SYSTEM,

messages=[{"role": "user", "content": code}],variable text last

)

response.usage.cache_creation_input_tokenspaid at 1.25x

response.usage.cache_read_input_tokenspaid at 0.1x

From Lesson 11.15, Lesson 11.11

Numbers that matter#

Cache write: 1.25x base for 5 min TTL. 2x base for 1 hour TTL.
Cache read: $0.30/M against $3.00/M fresh. Ten times cheaper.
Break-even: 1 write and 1 read is 0.675x. 1 write and 10 reads is 0.205x.
Reuse rule: cache a block that you read 3 times or more in the TTL.
Batch: 50% off input and output. 24 h guaranteed, 2 to 6 h typical.
Batch plus cache: about 10% of the sync uncached bill.
Model routing: saves 40 to 70% of model cost.
All levers stacked: about 5 to 10% of the naive baseline.

From Lesson 11.15, Lesson 17.14, Lesson 17.15, Lesson 17.27, Lesson 11.11

Budget guards in order#

From Lesson 17.27, Lesson 11.11, 11.11 code/caching_cost.py

Two cache layers#

From Lesson 17.14, Lesson 11.11

Provider prompt caching#

Factor	Anthropic	OpenAI	Gemini
API style	`cache_control` markers	Automatic prefix	`CachedContent` API
Hit price	90% off input	50% off input	About 25% of normal
Write cost	25% surcharge	None	Storage per token·hour
Default TTL	5 min, 1 hour option	Up to 1 hour	User-set, 1 hour
Min prefix	1,024 tokens, 2,048 Haiku	1,024 tokens	4,096 Flash, 32,768 Pro
Use when	Stable 10k+ prompt, many turns	Serverless, no custom infra	Multi-day reuse of a corpus

From Lesson 11.15

Defaults in the code#

Setting	Default	Note
`similarity_threshold`	0.85	`SemanticCache`. The demo uses 0.75
`ttl_seconds`	3600	Exact and semantic caches
`max_size`	1000	`ExactCache`. Evicts the oldest entry
`temperature`	0.0	Above 0 the exact cache skips
`monthly_budget`	1000.0	`CostTracker` alert base
`spend_cap_multiplier`	2.0	Cap is contract times this
`kill_z_score`	4.0	Pauses the tenant above it
Anthropic TTL	5 min	`"ttl": "1h"` for one hour

From 11.11 code/caching_cost.py, 17.27 code/main.py, Lesson 11.15

Submit a batch#

requests.append({

"custom_id": f"request-{i}",match results to inputs

"method": "POST", "url": "/v1/chat/completions",

"body": {"model": "<model>", "messages": [{"role": "user", "content": query}]},

})

batch_file = client.files.create(file=open("batch_input.jsonl", "rb"), purpose="batch")

batch = client.batches.create(

input_file_id=batch_file.id,

endpoint="/v1/chat/completions",

completion_window="24h",50% off, results in 24 hours

)

From Lesson 11.11

Troubleshooting#

Symptom	Cause	Fix
Hit rate near 7%	Time or request ID in the prefix	Put dynamic text after the boundary
Parallel calls: N writes	Calls beat the first write	Send request 1 alone, then the rest
`cache_read_input_tokens` stays 0	Cache keys drift	Check both usage fields in CI
Block does not cache	Under 1,024 tokens (2,048 Haiku)	Make the cached block larger
OpenAI hits stop on deploy	Tool reorder or new `user` field	Serialize tools in a stable order
`cached exceeds input`	Cached counted as extra tokens	Cached is a subset of input

From Lesson 17.14, Lesson 11.15, Lesson 11.11, token-counter-and-cost-meter/solution

Run the code#

Command	What it does
`python main.py`	Runs the lesson simulator in `code/`
`python3 scripts/project_test.py <project> --stage 1 --path <workspace>`	Grades one stage
`python3 scripts/project_test.py <project> --all --solution --strict`	Grades the reference solution
`python3 <workspace>/usage.py <usage-file> --out usage-ledger.json`	Replays recorded usage into a ledger
`python3 cli.py samples/input.json --mode execute --output budget.json`	Runs budget planner jobs

From 11.15 code/main.py, token-counter-and-cost-meter/README.md, agent-budget-planner/README.md, solution/cli.py

Project ledgers#

Rates: integer nano-dollars per token. Fixture inputs, not provider prices.
estimate: text.chars().count() over chars_per_token, rounded up. Ratio 0 is an error.
cost: usage.input - usage.cached at the input rate, plus cached and output. Each uses checked_mul.
reserve: spent + sum(holds) + amount > limit raises budget exceeded.
settle: actual above the hold raises actual cost exceeds reservation.
Statuses: completed, rejected, needs_reconciliation.
Sample run: 190 units against a 100-unit limit. Two jobs pass. 95 units settle.

From token-counter-and-cost-meter/solution, token-counter-and-cost-meter/README.md, solution/main.py, solution/cli.py, agent-budget-planner/README.md

Q&A#

Prompt or semantic cache?: Prompt: same tokens. Semantic: same meaning.
Which semantic threshold?: 0.92 to 0.95. At 0.85 wrong answers appear.
Which unit metric?: Cost per resolved query, not $/M tokens.

From Lesson 11.15, Lesson 11.11, Lesson 17.14, Lesson 17.15, Lesson 17.27

Tips#

Stamp user_id, task_id, route and tenant_id on every call.
Split tokens: prompt, tool, memory, response.
Gate deploys on read fraction above 80%.

From Lesson 17.27, Lesson 11.15