Using inference logging to check a deployed RAG application's performance

Databricks Model Serving supports inference tables that automatically log requests and responses for models deployed to a serving endpoint, including RAG applications built with Databricks. These logs capture the actual inputs (user queries), outputs (generated responses), and metadata over time, providing the raw data needed to evaluate a deployed RAG application's real-world performance and detect drift or quality issues after launch.

1 · Learn the must-know

• 1Inference tables must be enabled on the serving endpoint at creation or update time; they are not on by default and require a Unity Catalog-enabled workspace.
• 2Logged inference data is written as a Delta table in Unity Catalog, so it can be queried with SQL and joined with other tables (e.g., ground-truth or feedback tables) for analysis.
• 3Inference tables capture request and response payloads along with timestamps and status codes, but do not automatically capture intermediate RAG artifacts like retrieved chunks unless the application logs those explicitly (e.g., via MLflow tracing) as part of the response payload.
• 4There can be a delay between when a request hits the endpoint and when it appears in the inference table, so near-real-time monitoring should account for this lag rather than expecting instant availability.
• 5Because inference tables record production traffic, they are the recommended source for building ongoing evaluation and monitoring pipelines (e.g., scheduled jobs using Databricks' evaluation tooling) rather than relying solely on pre-deployment offline evaluation.
• 6Sensitive or PII-containing queries and responses are logged as-is, so governance controls (Unity Catalog permissions, table ACLs) should be applied to inference tables just like any other table containing production data.

2 · Check your understanding