Choosing an LLM's size and architecture from quantitative evaluation metrics

When choosing an LLM for a GenAI application, engineers must balance quality metrics against cost, latency, and throughput requirements rather than picking the largest or newest model by default. Databricks recommends benchmarking candidate models on task-specific offline evaluation metrics using representative evaluation datasets before committing to production, and revisiting the choice as requirements or available models change.

1 · Learn the must-know

• 1Quantitative comparison should combine quality metrics (e.g., accuracy, F1, ROUGE/BLEU for summarization/translation, or LLM-judged relevance/correctness) with operational metrics like latency (time to first token, tokens/sec) and cost per token/request.
• 2Larger models generally yield higher quality on complex reasoning tasks but increase inference latency and cost, so the objective is to find the smallest/cheapest model that still meets the required quality bar for the specific task.
• 3Architecture choice matters: a general-purpose foundation model may need fine-tuning or a smaller specialized/instruction-tuned model may outperform a larger general model on a narrow task at lower cost.
• 4Mosaic AI Model Serving supports deploying and comparing multiple model endpoints (including external models, foundation models, and custom/fine-tuned models) so teams can run head-to-head evaluations under production-like conditions.
• 5MLflow Evaluate (mlflow.evaluate) can be used to run the same evaluation dataset and metric set across multiple candidate models/endpoints to produce comparable quantitative results before selecting a model.
• 6A common gotcha: optimizing purely for a single quality metric can be misleading—always evaluate against multiple metrics and real business constraints (cost, latency, context window, throughput) since the best-scoring model on one metric may be impractical to serve at scale.

2 · Check your understanding