Folding subject-matter-expert feedback back into agent performance

Subject Matter Expert (SME) feedback closes the loop between offline evaluation and real-world quality by grounding agent improvements in domain expertise rather than developer intuition alone. In Databricks, this typically means collecting structured feedback (ratings, corrections, pass/fail judgments) on agent outputs and using it to refine prompts, retrieval logic, evaluation criteria, or fine-tuning data. Mosaic AI Agent Evaluation and the Review App are the primary mechanisms for capturing and operationalizing this feedback.

1 · Learn the must-know

• 1The Databricks Review App lets SMEs interact with an agent and label responses (e.g., thumbs up/down, free-text comments, or edits) without needing engineering access.
• 2SME feedback captured through the Review App is logged to Delta tables (via Lakehouse Monitoring/Inference tables), enabling traceability and downstream analysis.
• 3SME-labeled examples can be used to build or expand evaluation datasets, improving the relevance of automated LLM-judge metrics for a specific domain.
• 4Disagreement between SME judgments and LLM-judge scores should be reconciled by adjusting judge prompts or grading criteria, not by discarding SME input, since SMEs represent ground truth for domain correctness.
• 5Feedback should be iterative and versioned: track which prompt/retrieval/model version produced the response being reviewed so improvements can be attributed correctly.
• 6A common pitfall is treating SME feedback as a one-time collection exercise rather than an ongoing cycle integrated into the agent development and monitoring lifecycle.

2 · Check your understanding