Landing data with Auto Loader, and handling schema enforcement and evolution

Auto Loader (cloudFiles) incrementally and efficiently processes new files landing in cloud storage, using either directory listing or file notification mode to discover files. It enforces a stored schema to prevent bad data from corrupting a table, and supports schema evolution to safely adapt when new columns appear, all while writing into Unity Catalog-governed managed or external tables.

1 · Learn the must-know

• 1Auto Loader uses cloudFiles as the streaming source format, with cloudFiles.format specifying the underlying file type (json, csv, parquet, etc.).
• 2Directory listing mode incrementally lists cloud storage directories to detect new files and requires no extra cloud service setup, making it the simpler default option.
• 3File notification mode uses cloud provider queue/notification services (e.g., subscribing to storage event notifications) to detect new files, scaling better for very large or high-volume directories but requiring additional cloud resource permissions.
• 4Auto Loader infers and stores schema in a schema location, and cloudFiles.inferColumnTypes plus a checkpoint/schema directory let it track schema across runs so it does not re-infer every batch.
• 5cloudFiles.schemaEvolutionMode controls behavior when new columns are detected: default mode fails the stream and requires a restart to pick up the new schema (addNewColumns), while other modes can rescue data, ignore new columns, or fail explicitly.
• 6Unrecognized or mismatched columns are captured in a rescued data column (by default _rescued_data) rather than silently dropped, preserving data integrity.
• 7Even though Auto Loader is a streaming source, it can be run in batch/triggered fashion using trigger(availableNow=True) (or Trigger.Once) to process all currently available new files and then stop, which is the pattern typically used to land data incrementally into Unity Catalog tables via writeStream.table() or equivalent.

2 · Check your understanding