Deduplicating and aggregating DataFrames
Deduplication in Spark DataFrames uses dropDuplicates() (optionally on specific columns) or distinct(), and both are wide transformations requiring a shuffle. Aggregation functions like count, approx_count_distinct, mean/avg, and summary/describe let you profile and summarize data, with approx_count_distinct trading exact accuracy for speed on large datasets.
Must-know
- dropDuplicates() without arguments compares all columns; dropDuplicates(["col1","col2"]) dedupes based on a subset, keeping an arbitrary remaining row per group.
- distinct() is equivalent to dropDuplicates() with no column arguments, considering the entire row.
- count() returns the exact number of rows/non-null values and can be expensive on very large datasets since it requires a full scan.
approx_count_distinct() uses the HyperLogLog algorithm to estimate distinct counts much faster than countDistinct(), at the cost of a small, configurable error rate (default ~5%).- summary() returns count, mean, stddev, min, max, and percentiles (25%, 50%, 75%) by default and accepts custom statistics as arguments, unlike describe() which only gives count, mean, stddev, min, max.
- groupBy().agg() combined with functions like count, mean, sum, min, max is the standard pattern for computing aggregate statistics per group, and null values are excluded from numeric aggregations like mean by default.
A data engineer needs to remove duplicate customer records from customers_df, but only wants to keep the first occurrence based on customer_id, ignoring differences in other columns like last_updated. Which code accomplishes this?
What you have tried across Databricks DEA's objectives, not a readiness score.
Databricks Intelligence Platform
Data Ingestion and Loading
- Batch, streaming, and incremental loading patterns, and where the data comes from
- Loading files from cloud storage into governed tables with COPY INTO
- Landing data with Auto Loader, and handling schema enforcement and evolution
- Setting up Lakeflow Connect to ingest from enterprise sources reliably
- Pulling data through JDBC, ODBC, or REST clients and scheduling the job
- Choosing the right ingestion method for a given volume, frequency, and governance need
- Bringing semi-structured and unstructured data into governed Delta tables
Data Transformation and Modeling
- Cleaning bronze data into silver tables with PySpark and SQL
- Joining and combining DataFrames with the different join and union types
- Reshaping columns, rows, and arrays in a table
- Deduplicating and aggregating DataFrames
- Tuning Spark's core parameters and measuring what changed
- Building Gold-layer views and tables for BI and analytics
- Validating Silver and Gold datasets for quality
Working with Lakeflow Jobs
Implementing CI/CD
- Branching, committing, and opening pull requests from inside the Databricks workspace
- Promoting one codebase across dev, test, and prod with bundle variables and overrides
- Packaging and deploying jobs and pipelines with Automation Bundles
- Validating and managing bundle deployments from the Databricks CLI
Troubleshooting, Monitoring, and Optimization
- Spotting performance trends in a job's run history
- Reading job status, task graphs, and failure rates to monitor pipeline health
- Diagnosing skew, shuffle, and spill from Spark UI stage metrics
- What Liquid Clustering and predictive optimization actually do
- Diagnosing cluster startup failures, library conflicts, and out-of-memory errors
Governance and Security
Coverage checked against the published exam guide on Jul 27, 2026.
These are independent practice questions, written against this certification's published exam guide. They are not the certification vendor's own questions, and not the real exam.