Spotting batch inference workloads and applying ai_query() to them

Batch inference means scoring large sets of records offline (e.g., in a Delta table) rather than serving real-time requests one at a time. Databricks lets you run this directly in SQL or Spark using the ai_query() function, which invokes a model serving endpoint against each row of a table. This avoids building custom pipelines for calling LLMs or ML models over large datasets.

1 · Learn the must-know

• 1ai_query() is a built-in SQL function that sends each row of a query as input to a specified model serving endpoint and returns the model's response as a column, making it ideal for batch scoring within Databricks SQL, notebooks, or jobs.
• 2It works with both foundation model endpoints (e.g., for embeddings, chat/completions) and custom or fine-tuned model serving endpoints, as long as the endpoint accepts the request format ai_query() sends.
• 3Because it runs at scale over Spark/SQL, ai_query() is well suited for batch inference workloads like classifying large tables, generating embeddings for RAG pipelines, or summarizing many documents at once—not for low-latency, single-record real-time serving.
• 4You should be mindful of endpoint throughput/concurrency limits and cost, since applying ai_query() over millions of rows can generate a very large number of endpoint calls.
• 5ai_query() can be embedded directly in a Delta Live Tables or Lakeflow pipeline, or scheduled as part of a Databricks job, enabling repeatable batch inference workflows.
• 6Unlike real-time model serving (used for interactive apps needing millisecond responses), batch inference via ai_query() is appropriate when latency is not critical and throughput over large datasets matters more.

2 · Check your understanding