Choosing Parquet, Delta, CSV, JSON, or Iceberg as the table format

In Azure Databricks, the choice of data format affects performance, schema handling, and transactional guarantees at each stage of the data pipeline. Delta Lake, built on Parquet, is the recommended default for most workloads because it adds ACID transactions, schema enforcement, and time travel, while raw formats like CSV, JSON, and plain Parquet remain common for ingestion (bronze) layers.

1 · Learn the must-know

• 1Delta Lake stores data as Parquet files plus a transaction log (_delta_log), enabling ACID transactions, schema enforcement/evolution, time travel, and efficient upserts via MERGE that raw Parquet, CSV, and JSON cannot provide.
• 2CSV and JSON are typically used only for raw/bronze ingestion because they are row-based, lack built-in schema enforcement, and are slower to query than columnar formats like Parquet or Delta.
• 3Plain Parquet offers efficient columnar storage and compression but has no built-in transaction log, so concurrent writes, upserts, and deletes are unsafe or unsupported without an additional layer like Delta.
• 4Databricks Runtime uses Delta as the default table format when you run CREATE TABLE without specifying USING, so explicit format selection (e.g., USING CSV, USING PARQUET) is required to override this default.
• 5Apache Iceberg is an open table format that Databricks can read/write via Delta Lake UniForm (Universal Format) or Unity Catalog managed tables, allowing interoperability with external engines without duplicating data.
• 6For medallion architecture, Microsoft recommends converting raw CSV/JSON bronze data into Delta tables as early as possible to gain reliability, performance, and governance benefits in silver and gold layers.

2 · Check your understanding