Profiling data for summary statistics and distributions

In Azure Databricks, you can quickly profile a DataFrame to generate summary statistics and visualize data distributions using built-in notebook tools and Spark SQL functions. This helps identify data quality issues, skew, nulls, and outliers before further transformation or modeling.

1 · Learn the must-know

• 1Calling display() on a DataFrame in a Databricks notebook shows a 'Data Profile' tab that automatically computes summary statistics (count, mean, stddev, min, max, percentiles) and histograms for each column.
• 2The dbutils.data.summarize(df) command produces an interactive profiling report similar to the Data Profile tab, including numeric distributions, categorical value counts, and missing value percentages.
• 3Spark DataFrame's built-in .describe() method returns count, mean, stddev, min, and max for numeric and string columns, while .summary() extends this with configurable percentiles (e.g., 25%, 50%, 75%).
• 4Profiling large datasets can trigger a full data scan, so sampling (df.sample()) is recommended before profiling very large tables to control cost and runtime.
• 5Null and distinct value counts shown in the Data Profile tab help detect data quality problems such as missing values or unexpected cardinality in categorical columns.
• 6Data profiling results are not persisted automatically; if you need repeatable profiling reports, you should capture and store the summary output (e.g., write to a table or file) as part of a pipeline.

2 · Check your understanding