Cleaning bronze data into silver tables with PySpark and SQL

In the medallion architecture, silver tables clean and conform bronze data by removing nulls, deduplicating, and enforcing correct data types before further transformation. This step typically uses PySpark DataFrame operations or SQL to read from a bronze Delta table, apply cleaning logic, and write the result as a new managed or external Delta table.

1 · Learn the must-know

• 1Use dropna()/na.drop() or SQL WHERE col IS NOT NULL to remove nulls, and fillna()/na.fill() or COALESCE to impute default values instead.
• 2Use dropDuplicates() or SQL DISTINCT/ROW_NUMBER() window logic to remove duplicate records, since bronze data often contains raw, duplicated ingested events.
• 3Use cast() in PySpark or CAST()/:: in SQL to standardize data types (e.g., string to timestamp, string to decimal) since bronze tables often store all fields loosely typed or as strings.
• 4Write cleaned output with df.write.format('delta').mode('overwrite'|'append').saveAsTable() or CREATE TABLE ... AS SELECT (CTAS) to persist the silver table.
• 5Column pruning and renaming (select, withColumnRenamed, alias) commonly happen at this stage to conform to a clean, documented schema for downstream consumers.
• 6Schema enforcement is automatic with Delta Lake, so mismatched types on write will throw an error unless mergeSchema or overwriteSchema options are explicitly set.

2 · Check your understanding