Ingesting from a change data capture feed

Change Data Feed (CDF) is a Delta Lake feature that records row-level INSERT, UPDATE, and DELETE operations on a table, enabling downstream consumers to incrementally process only what changed rather than rescanning full tables. In Azure Databricks, CDF is commonly paired with Delta Live Tables' APPLY CHANGES INTO (SQL) or apply_changes (Python) to build CDC ingestion pipelines that keep target tables in sync with changing source data, including SCD Type 1 and Type 2 patterns.

1 · Learn the must-know

• 1CDF must be explicitly enabled on a Delta table via the table property delta.enableChangeDataFeed = true, either at CREATE TABLE time or with ALTER TABLE ... SET TBLPROPERTIES.
• 2Enabled CDF exposes change rows with metadata columns _change_type (insert, update_preimage, update_postimage, delete), _commit_version, and _commit_timestamp, readable via the readChangeFeed option in batch or streaming reads with startingVersion/startingTimestamp.
• 3Delta Live Tables' APPLY CHANGES INTO (SQL) or apply_changes (Python) is the recommended way to ingest a CDC feed into a target streaming table, automatically handling inserts, updates, and deletes.
• 4APPLY CHANGES requires a KEYS clause to identify matching rows and a SEQUENCE BY column (typically a timestamp or version) to correctly order out-of-sequence change records.
• 5The STORED AS SCD TYPE 1 or SCD TYPE 2 option controls whether history is overwritten (Type 1) or preserved with effective/end dates and additional tracking columns (Type 2).
• 6CDF (Delta's internal change tracking) is distinct from ingesting CDC data emitted by external source systems (e.g., staged insert/delete/update records from a database); APPLY CHANGES can consume either kind of change feed as long as it has keys and a sequencing column.

2 · Check your understanding