Production Observability, Latency, and Cost
A green API call can still be a failed task. Separate system reliability from task quality and business outcome. Design logs, metrics, and traces for Claude requests and agent trajectories. Diagnose latency across model, retrieval, tools, queues, and retries. Measure total cost and cost per successful outcome. Define alerts and rollout gates from service objectives. A production dashboard reports 99.9 percent successful API calls. Customers are still complaining. The model returns HTTP 200, but some answers use stale sources. A tool times out and the agent silently continues. Long prompts miss the cache because a timestamp was placed near the beginning. P95 latency has doubled while the average looks acceptable. A cheaper model lowered call price but increased retries and human review. The dashboard measures transport success. The product depends on task success. Observability must connect the two. System signals tell you whether components ran. Trajectory signals tell you what the application did. Quality signals tell you whether the result met the task. Business signals tell you whether the workflow created value. Do not collapse them into one "success" field. Logs record discrete events: request accepted, retrieval returned no candidates, tool rejected authorization, output failed schema validation, reviewer escalated. Use structured fields so operators can group and filter them. Metrics aggregate behavior over time: request rate, error rate, P95 latency,…
Production Observability, Latency, and Cost: A green API call can still be a failed task. Separate system reliability from task quality and business outcome.…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.