Deduplicating and aggregating DataFrames

Deduplication in Spark DataFrames uses dropDuplicates() (optionally on specific columns) or distinct(), and both are wide transformations requiring a shuffle. Aggregation functions like count, approx_count_distinct, mean/avg, and summary/describe let you profile and summarize data, with approx_count_distinct trading exact accuracy for speed on large datasets.

1 · Learn the must-know

• 1dropDuplicates() without arguments compares all columns; dropDuplicates(["col1","col2"]) dedupes based on a subset, keeping an arbitrary remaining row per group.
• 2distinct() is equivalent to dropDuplicates() with no column arguments, considering the entire row.
• 3count() returns the exact number of rows/non-null values and can be expensive on very large datasets since it requires a full scan.
• 4approx_count_distinct() uses the HyperLogLog algorithm to estimate distinct counts much faster than countDistinct(), at the cost of a small, configurable error rate (default ~5%).
• 5summary() returns count, mean, stddev, min, max, and percentiles (25%, 50%, 75%) by default and accepts custom statistics as arguments, unlike describe() which only gives count, mean, stddev, min, max.
• 6groupBy().agg() combined with functions like count, mean, sum, min, max is the standard pattern for computing aggregate statistics per group, and null values are excluded from numeric aggregations like mean by default.

2 · Check your understanding