Joining and combining DataFrames with the different join and union types

Spark SQL and DataFrame APIs support standard SQL join types plus union operations for combining datasets. Join type and key selection affect result cardinality and null handling, while broadcast joins optimize performance when one side is small.

Must-know

• Inner join returns only matching rows from both DataFrames; left (outer) join keeps all rows from the left side, filling unmatched right-side columns with null.
• Multiple join keys are specified by passing a list of column names or an AND-combined boolean expression, e.g. df1.join(df2, ['col1','col2']) or df1.join(df2, (df1.a==df2.a) & (df1.b==df2.b)).
• Broadcast join (broadcast() hint or automatic via spark.sql.autoBroadcastJoinThreshold) sends a small DataFrame to all executors to avoid a costly shuffle when joining with a much larger table.
• Cross join produces the Cartesian product of two DataFrames (every row paired with every row) and can explode row counts quickly, so it must be used deliberately, often via crossJoin() or an explicit CROSS JOIN.
• union() (or unionByName()) combines rows from two DataFrames with the same schema and does NOT remove duplicates, unlike SQL UNION which dedupes; use distinct() after union to mimic that behavior.
• unionByName() matches columns by name rather than position, which avoids silently misaligned data when column order differs between DataFrames, and can optionally allow missing columns with allowMissingColumns=True.

Coverage checked against the published exam guide on Jul 25, 2026.

These are independent practice questions, written against this certification's published exam guide. They are not the certification vendor's own questions, and not the real exam.