Setting CPU, node count, autoscaling, termination, node type, cluster size, and pooling

Azure Databricks compute configuration lets you control cost and performance by choosing node types, cluster sizing, autoscaling, and termination policies when creating clusters. For the Data Engineer Associate exam, focus on how driver/worker node types, autoscaling min/max workers, auto-termination, and pools affect cluster startup time, elasticity, and cost.

1 · Learn the must-know

• 1A cluster's node type determines the VM family (CPU, memory, and optionally GPU) for both the driver and worker nodes, and larger/more memory-optimized nodes are chosen for shuffle-heavy or large-data workloads.
• 2Autoscaling lets you set a minimum and maximum number of worker nodes; Databricks automatically adds workers under load and removes them when idle, which optimizes cost but can add latency during scale-up events.
• 3Auto-termination (inactivity timeout) automatically shuts down an all-purpose cluster after a specified number of idle minutes to avoid paying for unused compute, but it does not apply to job clusters, which terminate automatically when the job completes.
• 4Cluster pools pre-provision and keep a set of idle, ready-to-use VM instances so that new clusters attached to the pool can start faster and skip cloud provisioning time, reducing cold-start latency (though you still pay Azure VM costs for idle pooled instances, not DBU costs).
• 5Choosing between a fixed-size cluster and an autoscaling cluster is a trade-off: fixed-size gives predictable performance for consistent workloads, while autoscaling suits variable or unpredictable workloads but requires setting sensible min/max bounds.
• 6Job clusters (created and terminated per job run) versus all-purpose/interactive clusters (persistent, shared) differ in lifecycle and termination behavior, which affects cost management and is a common exam distinction.

2 · Check your understanding