Tuning Spark's core parameters and measuring what changed

Spark performance tuning on Databricks centers on a handful of configuration parameters that control shuffle behavior, memory allocation, and join strategy. Learners should know default values, when to adjust them, and how to validate improvements by re-running and comparing query metrics (e.g., via the Spark UI or query execution time) after each change.

Must-know

• spark.sql.shuffle.partitions controls the number of partitions used when shuffling data for joins or aggregations; it defaults to 200 and often needs to be lowered for small datasets or raised for large clusters to avoid too many small tasks or too few large ones.
• spark.default.parallelism sets the default number of partitions for RDD operations (not DataFrame/SQL operations, which use spark.sql.shuffle.partitions instead), so it has limited effect in typical DataFrame-based Databricks workloads.
• spark.executor.memory and spark.driver.memory set the JVM heap size for executors and the driver respectively; increasing them can prevent out-of-memory errors and spill-to-disk during large shuffles or aggregations, but over-allocating can reduce the number of executors that fit on a cluster.
• spark.sql.autoBroadcastJoinThreshold determines the max size (default 10MB) of a table that Spark will automatically broadcast to all executors for a broadcast hash join, avoiding an expensive shuffle join; setting it to -1 disables auto-broadcasting entirely.
• Tuning is iterative: change one parameter at a time, re-run the workload, and compare metrics (execution time, shuffle read/write, spill) in the Spark UI to confirm the change actually helped rather than assuming improvement.
• These are session- or cluster-level configs typically set via spark.conf.set() or cluster configuration, and changes only apply to queries/jobs run after the setting is applied, not retroactively.

Coverage checked against the published exam guide on Jul 26, 2026.

These are independent practice questions, written against this certification's published exam guide. They are not the certification vendor's own questions, and not the real exam.