Diagnosing cluster startup failures, library conflicts, and out-of-memory errors

Cluster failures fall into three buckets: startup issues (cloud resource limits, init scripts, network/permissions), library conflicts (dependency version clashes or scope mismatches), and out-of-memory errors (driver vs executor memory pressure). Diagnosis relies on reading cluster event logs, driver/executor logs, and Ganglia/metrics UI (or the newer cluster metrics tab) rather than guessing.

1 · Learn the must-know

• 1Cluster startup failures often show as 'Pending' then terminate; check the Event Log tab first for reasons like cloud provider quota limits, invalid instance types, or failed init scripts before checking logs.
• 2Init script failures are a common startup cause; script output/errors are captured in cluster logs (DBFS or cloud storage logging destination) and should be checked line by line.
• 3Library conflicts typically arise from mixing cluster-installed (init script/UI) libraries with notebook-scoped (%pip, %conda) libraries, or from incompatible versions across the same cluster; notebook-scoped installs affect only the attached notebook's REPL.
• 4Driver out-of-memory usually results from collect(), toPandas(), or broadcasting large datasets to the driver, which has limited memory compared to executors; the fix is to avoid pulling large data to the driver or increase driver node size.
• 5Executor OOM often stems from data skew, overly large partitions, or insufficient shuffle partitions; repartitioning, salting skewed keys, or increasing executor memory/nodes are standard remedies.
• 6The Spark UI (Storage, Executors, and SQL tabs) and cluster metrics are the primary tools to confirm memory pressure, spill to disk, or GC overhead before resizing a cluster or changing code.

2 · Check your understanding