Picking the key metrics to monitor for a given LLM deployment

Choosing the right monitoring metrics depends on matching the metric type to the LLM deployment's business goal, whether that's output quality, cost efficiency, latency, or safety. Databricks recommends combining operational metrics (latency, throughput, token usage, error rates) with quality metrics (relevance, correctness, toxicity, groundedness) rather than relying on any single measure. The specific scenario—chatbot, RAG system, summarization, classification—determines which quality dimensions matter most.

1 · Learn the must-know

• 1For latency-sensitive applications like real-time chatbots, monitor time-to-first-token and total response latency, not just throughput.
• 2Cost monitoring should track token consumption (input/output separately) since pricing and context limits scale with token count, not just request count.
• 3For RAG deployments, monitor retrieval quality (relevance of retrieved chunks) separately from generation quality (faithfulness/groundedness to retrieved context) since failures can occur at either stage.
• 4Toxicity, bias, and PII leakage metrics are critical for customer-facing deployments even if the primary use case is unrelated to safety.
• 5A single aggregate quality score can mask problems; segment metrics by input type, user cohort, or time window to catch drift or edge-case failures.
• 6Business/task-specific metrics (e.g., resolution rate, click-through, task completion) should be tracked alongside generic LLM metrics since high fluency or low toxicity doesn't guarantee the model is achieving its intended purpose.

2 · Check your understanding