Ingesting with Lakeflow Spark Declarative Pipelines and Auto Loader

Lakeflow Spark Declarative Pipelines let you define ETL as declarative streaming tables and materialized views in SQL or Python, and Auto Loader (the cloudFiles source) is the standard way to incrementally ingest new files from cloud object storage into these pipelines. Together they provide scalable, checkpointed, schema-aware ingestion without manually tracking which files have already been processed.

1 · Learn the must-know

• 1Auto Loader is invoked with format('cloudFiles') and requires cloudFiles.format to specify the underlying file type (e.g., json, csv, parquet, avro).
• 2Declarative pipelines typically use CREATE OR REFRESH STREAMING TABLE (SQL) or @dlt.table with spark.readStream (Python) to wire an Auto Loader source into the pipeline's managed DAG.
• 3Auto Loader supports directory listing (default) and file notification modes; file notification uses cloud-native queue/event services to scale to very large numbers of files with lower listing overhead.
• 4Schema inference and evolution are controlled by cloudFiles.schemaEvolutionMode (e.g., addNewColumns, rescue, failOnNewColumns, none), and unmatched data is captured in a rescued data column (_rescued_data) by default.
• 5Auto Loader relies on a checkpoint location to track which files have been ingested, guaranteeing exactly-once processing even after pipeline restarts.
• 6Within a pipeline, expectations (constraints) can be attached to streaming tables fed by Auto Loader to validate, drop, or fail records that don't meet data quality rules during ingestion.

2 · Check your understanding