Landing data with Auto Loader, and handling schema enforcement and evolution

Auto Loader (cloudFiles) incrementally and efficiently processes new files landing in cloud storage, using either directory listing or file notification mode to discover files. It enforces a stored schema to prevent bad data from corrupting a table, and supports schema evolution to safely adapt when new columns appear, all while writing into Unity Catalog-governed managed or external tables.

Must-know

• Auto Loader uses cloudFiles as the streaming source format, with cloudFiles.format specifying the underlying file type (json, csv, parquet, etc.).
• Directory listing mode incrementally lists cloud storage directories to detect new files and requires no extra cloud service setup, making it the simpler default option.
• File notification mode uses cloud provider queue/notification services (e.g., subscribing to storage event notifications) to detect new files, scaling better for very large or high-volume directories but requiring additional cloud resource permissions.
• Auto Loader infers and stores schema in a schema location, and cloudFiles.inferColumnTypes plus a checkpoint/schema directory let it track schema across runs so it does not re-infer every batch.
• cloudFiles.schemaEvolutionMode controls behavior when new columns are detected: default mode fails the stream and requires a restart to pick up the new schema (addNewColumns), while other modes can rescue data, ignore new columns, or fail explicitly.
• Unrecognized or mismatched columns are captured in a rescued data column (by default _rescued_data) rather than silently dropped, preserving data integrity.
• Even though Auto Loader is a streaming source, it can be run in batch/triggered fashion using trigger(availableNow=True) (or Trigger.Once) to process all currently available new files and then stop, which is the pattern typically used to land data incrementally into Unity Catalog tables via writeStream.table() or equivalent.

Coverage checked against the published exam guide on Jul 26, 2026.

These are independent practice questions, written against this certification's published exam guide. They are not the certification vendor's own questions, and not the real exam.