Evaluating retrieval performance with the right tools and metrics

Retrieval quality directly bounds RAG answer quality, so it must be evaluated separately from generation quality. Databricks recommends evaluating retrievers using standard information retrieval metrics computed over a labeled evaluation set of (query, relevant document/chunk) pairs, and provides tooling such as Mosaic AI Agent Evaluation to help compute and track these metrics.

1 · Learn the must-know

• 1Precision@K measures what fraction of the top-K retrieved chunks are actually relevant, penalizing irrelevant results returned to the LLM context.
• 2Recall@K measures what fraction of all known relevant chunks were successfully retrieved in the top-K, indicating whether important context is being missed.
• 3NDCG (Normalized Discounted Cumulative Gain) accounts for the ranking order of relevant results, rewarding retrievers that place the most relevant chunks higher.
• 4MRR (Mean Reciprocal Rank) focuses on how quickly the first relevant result appears, useful when only one correct answer/chunk is expected.
• 5Building a labeled ground-truth dataset of representative queries mapped to known-relevant document chunks is required before these metrics can be computed, and Databricks supports curating such evaluation sets (including via human or LLM-assisted labeling) as part of the evaluation workflow.
• 6A common gotcha: retrieval metrics can look strong on a small or non-representative evaluation set yet fail to reflect real production performance, so evaluation sets should be periodically refreshed and reviewed for coverage.

2 · Check your understanding