Using the DAG, Spark UI, and query profile to fix cache, skew, spill, and shuffle

Performance problems in Spark on Azure Databricks—caching misuse, data skew, spill, and excessive shuffle—can be diagnosed by reading the DAG visualization, Spark UI stage/task metrics, and the SQL query profile. These tools expose task-level timing, data volumes, and partition imbalance so you can pinpoint the root cause before tuning code or cluster configuration.

1 · Learn the must-know

• 1The Spark UI's Stages tab shows per-task duration, shuffle read/write, and spill (memory/disk) metrics; a wide spread between min and max task duration or shuffle bytes indicates data skew.
• 2Spill occurs when a task's data exceeds available executor memory and Spark writes intermediate data to disk, visible as 'Spill (Memory)' and 'Spill (Disk)' columns in the Stages/Tasks UI, which signals a need for more memory, better partitioning, or reducing partition size skew.
• 3The DAG visualization (Jobs/Stages tab) shows stage boundaries caused by wide transformations (e.g., joins, groupBy, repartition) that trigger shuffles; unnecessary or repeated shuffles across stages point to opportunities for query rewrite or broadcast joins.
• 4The query profile (available for SQL/DataFrame queries, especially with Photon and the SQL warehouse UI) breaks down time spent per operator (scan, join, shuffle, aggregate) and flags skewed partitions and inefficient joins directly in a visual query plan.
• 5Caching (.cache()/.persist()) issues are diagnosed via the Storage tab, which shows whether a DataFrame is fully cached, partially cached (spilled to disk), or not cached at all, and whether cached data fits in the configured memory fraction.
• 6Adaptive Query Execution (AQE), enabled by default in Databricks Runtime, automatically coalesces shuffle partitions and can split skewed partitions at runtime, so check whether AQE is enabled before manually tuning shuffle partition counts or applying salting techniques.

2 · Check your understanding