Serving an LLM application through Foundation Model APIs

Databricks Foundation Model APIs let you query popular open (and select proprietary) LLMs through built-in, ready-to-use serving endpoints without deploying or managing your own model infrastructure. An LLM application can call these endpoints directly (pay-per-token) or via provisioned throughput for production-scale, low-latency needs, then compose the responses with other components like vector search or prompt templates.

1 · Learn the must-know

• 1Foundation Model APIs expose curated models on pay-per-token endpoints that require no capacity planning or deployment by the user, ideal for getting started or variable traffic.
• 2Provisioned throughput endpoints should be used for production workloads needing performance guarantees, higher throughput, or serving fine-tuned custom models.
• 3All Foundation Model API endpoints share a unified, OpenAI-compatible query format, so switching models often requires minimal code change.
• 4An LLM application built with Foundation Model APIs is still typically wrapped in its own Model Serving endpoint (e.g., via MLflow) if it adds custom logic such as chaining, retrieval, or business rules.
• 5External Models can be registered to route to third-party provider APIs through the same serving interface, letting an app mix foundation and external models behind one gateway.
• 6Rate limits and token costs differ between pay-per-token and provisioned throughput, so architects must choose based on expected request volume and latency SLAs before going to production.

2 · Check your understanding