RAG Evaluation: Precision, Recall, MRR, nDCG, Faithfulness, Answer Relevance
If you cannot grade your retrieval and your answer at the same time, you cannot ship the system. The two are not the same metric and the same prompt fails on different axes. Compute four retrieval metrics from gold qrels: precision@k, recall@k, MRR (mean reciprocal rank), and nDCG@k. Compute two answer-grade metrics: faithfulness (every claim grounded in retrieved context) and answer relevance (the answer addresses the question). Build a fixture qrels file (queries, gold doc ids, gold answer text) that the eval reads end to end. Read the metric values to diagnose where a pipeline is failing: retrieval, ranking, generation, or grounding. A RAG system has at least four moving parts: chunker, retriever, reranker, generator. Any of them can be the cause of a wrong answer. Without per-stage metrics you are flying blind. A user reports a wrong answer. Is it because the chunker cut the answer span? Is it because the retriever did not include the chunk in top-k? Is it because the reranker pushed the right chunk past position one? Is it because the generator ignored the chunk and made something up? You cannot tell from the answer alone. You need: Retrieval metrics to grade what came out of the retriever. Ranking metrics to grade where the right chunk sat in the order. Faithfulness to grade whether the…
RAG Evaluation: Precision, Recall, MRR, nDCG, Faithfulness, Answer Relevance: If you cannot grade your retrieval and your answer at the same time, you cannot…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.