Diagnosing skew, shuffle, and spill from Spark UI stage metrics

Spark UI stage-level metrics reveal where jobs slow down: task duration skew, shuffle read/write volume, and spill to disk all show up in the Stages tab's task summary and event timeline. Data skew appears as a small number of tasks taking far longer than the median, shuffling shows up as large shuffle read/write bytes, and spilling shows as memory spill and disk spill columns indicating a partition didn't fit in executor memory.

1 · Learn the must-know

• 1Data skew is diagnosed by comparing min/median/max task duration in the Stages tab; a max far above the median (long-tail tasks) indicates one or a few partitions hold disproportionate data.
• 2Shuffle read/write size and shuffle spill (memory) and shuffle spill (disk) are visible per-stage summary metrics; wide transformations (groupBy, join, distinct, repartition) trigger shuffles and are the usual root cause of skew and spill.
• 3Disk spilling occurs when a task's data exceeds available executor memory during a shuffle or aggregation and Spark writes intermediate data to disk, which sharply increases stage duration and I/O.
• 4Salting skewed join/group-by keys, using Adaptive Query Execution (AQE) to auto-optimize skewed joins and coalesce shuffle partitions, or increasing shuffle partition count are common fixes for skew and spill.
• 5The Spark UI SQL tab and stage DAG visualization help pinpoint which operator (join, aggregate, sort) caused the shuffle, complementing the raw stage metrics.
• 6A stage with many small tasks (over-partitioning) or very few large tasks (under-partitioning) both show up as inefficiency in the Stages tab and often correlate with skew or spill symptoms.

2 · Check your understanding