Choosing between job compute, serverless, warehouses, classic, and shared compute

Azure Databricks offers several compute types optimized for different workloads: all-purpose (interactive) clusters for exploration, job compute for automated pipelines, SQL warehouses for SQL analytics and BI, and serverless options that remove infrastructure management. Choosing the right compute type balances cost, performance, isolation, and use case (interactive development vs. scheduled production jobs vs. SQL querying).

1 · Learn the must-know

• 1Job compute clusters are created automatically when a job runs and terminate automatically when the job completes, making them more cost-effective than all-purpose clusters for scheduled/production workloads.
• 2All-purpose (classic) clusters are meant for interactive, collaborative work in notebooks and can be shared by multiple users simultaneously, but incur cost while running/idle unless auto-termination is configured.
• 3SQL warehouses (formerly SQL endpoints) are compute resources optimized specifically for running SQL queries via Databricks SQL, and come in classic, pro, and serverless variants.
• 4Serverless compute (available for both notebooks/jobs and SQL warehouses) removes the need to configure or manage cluster infrastructure, offering rapid startup and automatic scaling managed by Databricks.
• 5Shared/multi-user clusters allow several users to attach to the same compute resource simultaneously, which is efficient for teams but requires appropriate access mode configuration (e.g., shared access mode) to ensure proper data isolation and governance with Unity Catalog.
• 6Cluster access modes (e.g., single user vs. shared) affect which features (like certain library installs or specific language support) are available and impact Unity Catalog enforcement, so the access mode must match the intended use case.

2 · Check your understanding