Tracking a live LLM endpoint with inference tables and Agent Monitoring

Databricks Model Serving can log every request and response payload for a live LLM endpoint to a Delta table called an inference table, giving you a durable, queryable record of production traffic. Agent Evaluation / Agent Monitoring builds on these logs (or on trace data from MLflow tracing) to continuously score quality, safety, and performance metrics for deployed GenAI agents, surfacing them in a monitoring dashboard.

1 · Learn the must-know

• 1Inference tables must be enabled at endpoint creation/update time and are stored as managed Delta tables in Unity Catalog, with request and response payloads captured automatically for every call.
• 2Inference table logging is asynchronous and near real-time, so there can be a short delay between a request being served and the corresponding row appearing in the table.
• 3Agent Monitoring relies on MLflow tracing (spans) captured from the deployed agent to reconstruct multi-step reasoning, tool calls, and retrieval steps for evaluation, not just the raw input/output pair.
• 4Quality and safety metrics (e.g., relevance, groundedness, toxicity, guideline adherence) are computed using Databricks' built-in LLM judges, which can run on a sample or all of the logged production traffic on a schedule.
• 5Because inference tables can grow quickly and contain sensitive data, teams should apply table ACLs, retention/lifecycle policies, and column masking as needed rather than assuming default protection.
• 6Monitoring dashboards and metrics update on a schedule (not instantaneously), so newly deployed changes to an agent may take some time before their impact is reflected in the monitoring UI.

2 · Check your understanding