Enforcing schemas and handling schema drift

Delta Lake enforces schema on write by default, rejecting writes whose column names or types don't match the target table unless you explicitly opt in to evolution. Auto Loader extends this with configurable schema evolution modes and a rescued data column so that unexpected or drifting fields from source data don't silently break or get lost in ingestion pipelines.

1 · Learn the must-know

• 1By default, Delta Lake schema enforcement blocks appends/merges that add new columns, drop columns, or change data types incompatible with the target table, raising an AnalysisException.
• 2Setting the DataFrameWriter option mergeSchema to true (or the session setting spark.databricks.delta.schema.autoMerge.enabled) allows new columns to be appended to a Delta table's schema automatically during writes.
• 3Auto Loader's cloudFiles.schemaEvolutionMode option controls drift behavior, with values like addNewColumns (default, evolves schema and fails the stream once to restart with the new schema), rescue, failOnNewColumns, and none.
• 4Auto Loader automatically stores unexpected or mismatched fields in a _rescued_data column (JSON string) instead of failing or discarding the data, preserving it for later inspection.
• 5Schema hints (cloudFiles.schemaHints) let you enforce specific column types during inference/evolution without turning off schema enforcement entirely.
• 6overwriteSchema true is required (and destructive) when you intentionally replace a Delta table's schema, as opposed to mergeSchema which only adds compatible new columns.

2 · Check your understanding