Knowing which evaluation judges need a ground-truth label to run

Databricks evaluation judges (used with Mosaic AI Agent Evaluation / mlflow.evaluate) fall into two categories: those that need a labeled ground truth answer to compare against, and those that assess quality using only the input and generated output. Knowing which judges require ground truth is essential for correctly building evaluation datasets and interpreting metric availability.

1 · Learn the must-know

• 1Correctness (answer correctness) judges require a ground truth/expected answer because they measure factual agreement between the model output and the reference answer.
• 2Judges like answer relevance, groundedness/faithfulness, and context relevance do NOT require ground truth since they evaluate the output against the retrieved context or the input query alone.
• 3Ground truth is typically supplied as a labeled evaluation dataset column (e.g., an 'expected_response' or similar field) when calling mlflow.evaluate.
• 4If ground truth is omitted, ground-truth-dependent metrics such as correctness will be skipped or unavailable in the evaluation results.
• 5Judges that rely on retrieved context (e.g., groundedness, context sufficiency/relevance) instead depend on the retrieval step output, not a human-labeled answer.
• 6Choosing the right judges depends on whether curated reference answers exist; RAG quality checks (faithfulness, relevance) can run without them, but accuracy/correctness checks cannot.

2 · Check your understanding