Writing custom Scorers to evaluate agents and LLMs

Databricks Agent Evaluation (via MLflow) allows engineers to define custom scorers when the built-in judges do not cover a specific business requirement. Custom scorers let you encode deterministic logic (e.g., regex, JSON schema checks) or LLM-as-a-judge logic tailored to your use case, and they run alongside built-in metrics during mlflow.evaluate() runs. This is essential for evaluating agent outputs against criteria unique to your domain, such as tone, specific factual grounding, or compliance rules.

1 · Learn the must-know

• 1Custom scorers are typically implemented as Python functions decorated with @scorer (or passed as a list of custom metric objects) and registered in the evaluation call alongside standard Agent Evaluation judges.
• 2A custom scorer function receives structured inputs like the request, response, retrieved context, and expected/ground-truth fields, and must return a score (numeric, boolean, or categorical) often with a rationale string.
• 3Custom scorers can wrap an LLM as a judge (LLM-as-a-judge pattern) using a prompt template you control, giving flexibility beyond the built-in correctness/groundedness/relevance judges.
• 4Results from custom scorers appear in the MLflow evaluation results table and Databricks UI alongside built-in metrics, enabling side-by-side comparison across runs.
• 5Custom scorers must be deterministic or clearly documented if using an LLM judge, since inconsistent scoring logic undermines reproducibility of evaluation runs.
• 6A common gotcha is mismatching the expected input/output schema of the scorer function with the evaluation dataset columns, which causes evaluation failures or silently skipped rows.

2 · Check your understanding