Diagnosing skew, shuffle, and spill from Spark UI stage metrics

Spark UI stage-level metrics reveal where jobs slow down: task duration skew, shuffle read/write volume, and spill to disk all show up in the Stages tab's task summary and event timeline. Data skew appears as a small number of tasks taking far longer than the median, shuffling shows up as large shuffle read/write bytes, and spilling shows as memory spill and disk spill columns indicating a partition didn't fit in executor memory.

Must-know

• Data skew is diagnosed by comparing min/median/max task duration in the Stages tab; a max far above the median (long-tail tasks) indicates one or a few partitions hold disproportionate data.
• Shuffle read/write size and shuffle spill (memory) and shuffle spill (disk) are visible per-stage summary metrics; wide transformations (groupBy, join, distinct, repartition) trigger shuffles and are the usual root cause of skew and spill.
• Disk spilling occurs when a task's data exceeds available executor memory during a shuffle or aggregation and Spark writes intermediate data to disk, which sharply increases stage duration and I/O.
• Salting skewed join/group-by keys, using Adaptive Query Execution (AQE) to auto-optimize skewed joins and coalesce shuffle partitions, or increasing shuffle partition count are common fixes for skew and spill.
• The Spark UI SQL tab and stage DAG visualization help pinpoint which operator (join, aggregate, sort) caused the shuffle, complementing the raw stage metrics.
• A stage with many small tasks (over-partitioning) or very few large tasks (under-partitioning) both show up as inefficiency in the Stages tab and often correlate with skew or spill symptoms.

Coverage checked against the published exam guide on Jul 26, 2026.

These are independent practice questions, written against this certification's published exam guide. They are not the certification vendor's own questions, and not the real exam.