Tracking an LLM or agent through AI Gateway's inference tables, usage tables, and rate limiting

AI Gateway is a governance layer on Databricks Model Serving endpoints that lets you centrally manage access, cost, and observability for LLMs and agents deployed via Agent Framework. It provides Inference Tables for automatic logging of requests and responses, Usage Tables for tracking token consumption, and rate limiting to control traffic, all configurable without changing client code.

1 · Learn the must-know

• 1Inference Tables must be explicitly enabled on a serving endpoint and automatically capture request payloads, response payloads, and metadata (timestamp, status code) as a Delta table you can query for evaluation, monitoring, or debugging.
• 2Because Inference Tables are standard Delta tables, they can be joined with Lakehouse Monitoring or custom evaluation jobs to track drift, quality, or safety metrics over time.
• 3Usage Tables track token-level consumption (prompt and completion tokens) per endpoint, enabling cost attribution and budget monitoring across models or teams.
• 4Rate limiting in AI Gateway can be configured per endpoint (and in some cases per user or service principal) to throttle queries-per-minute or tokens-per-minute, protecting downstream systems and controlling spend.
• 5AI Gateway features apply uniformly whether the endpoint serves a foundation model, a fine-tuned model, or an agent built with Agent Framework, since agents are ultimately deployed as Model Serving endpoints.
• 6Enabling Inference Tables or rate limits requires endpoint configuration (via UI, API, or SDK) and may need permissions on the endpoint; changes do not require redeploying the underlying model.

2 · Check your understanding