Tuning Spark's core parameters and measuring what changed

Spark performance tuning on Databricks centers on a handful of configuration parameters that control shuffle behavior, memory allocation, and join strategy. Learners should know default values, when to adjust them, and how to validate improvements by re-running and comparing query metrics (e.g., via the Spark UI or query execution time) after each change.

1 · Learn the must-know

• 1spark.sql.shuffle.partitions controls the number of partitions used when shuffling data for joins or aggregations; it defaults to 200 and often needs to be lowered for small datasets or raised for large clusters to avoid too many small tasks or too few large ones.
• 2spark.default.parallelism sets the default number of partitions for RDD operations (not DataFrame/SQL operations, which use spark.sql.shuffle.partitions instead), so it has limited effect in typical DataFrame-based Databricks workloads.
• 3spark.executor.memory and spark.driver.memory set the JVM heap size for executors and the driver respectively; increasing them can prevent out-of-memory errors and spill-to-disk during large shuffles or aggregations, but over-allocating can reduce the number of executors that fit on a cluster.
• 4spark.sql.autoBroadcastJoinThreshold determines the max size (default 10MB) of a table that Spark will automatically broadcast to all executors for a broadcast hash join, avoiding an expensive shuffle join; setting it to -1 disables auto-broadcasting entirely.
• 5Tuning is iterative: change one parameter at a time, re-run the workload, and compare metrics (execution time, shuffle read/write, spill) in the Spark UI to confirm the change actually helped rather than assuming improvement.
• 6These are session- or cluster-level configs typically set via spark.conf.set() or cluster configuration, and changes only apply to queries/jobs run after the setting is applied, not retroactively.

2 · Check your understanding