Tuning Spark jobs and notebooks when they stall or run out of resources

Troubleshooting Spark jobs in Azure Databricks means diagnosing failures and slowness using the Spark UI, driver/executor logs, and cluster metrics, then applying fixes such as repartitioning, caching, or resizing clusters. Common root causes are data skew, spill to disk, small-file problems, and misconfigured executor/driver memory, all of which show up as long stages, high GC time, or shuffle-heavy plans in the Spark UI.

1 · Learn the must-know

• 1The Spark UI's Stages and SQL tabs reveal skewed tasks (a few tasks taking far longer than others), excessive shuffle read/write, and spill to disk, which are the top causes of slow jobs.
• 2Driver OOM errors often come from collect(), toPandas(), or broadcasting oversized tables, while executor OOM errors point to data skew or under-provisioned executor memory relative to partition size.
• 3Restarting a cluster clears cached data, temp views, and widget/notebook state, and is a valid fix for stuck jobs, corrupted metastore connections, library conflicts, or after changing cluster-scoped init scripts/libraries.
• 4Autoscaling and cluster sizing issues (too few workers, undersized instance types, or spot/preemption evictions) manifest as jobs that never speed up despite more data or as sporadic task failures needing driver logs to confirm.
• 5Small-file problems (many tiny files from frequent writes/streaming) slow reads and should be addressed with OPTIMIZE/compaction on Delta tables rather than just scaling the cluster.
• 6Repartition/coalesce, broadcast joins for small dimension tables, and adjusting shuffle partitions (spark.sql.shuffle.partitions) are standard tuning levers, but must be validated against the Spark UI rather than applied blindly.

2 · Check your understanding