Planning how data is extracted and which file types to ingest

Designing ingestion logic in Azure Databricks requires choosing the right extraction pattern (one-time batch vs. incremental/streaming) and matching it to the correct data source and file format so that Auto Loader, COPY INTO, or Spark readers can efficiently and reliably load data into Delta tables. The choice affects schema handling, scalability, and how new files are detected over time.

1 · Learn the must-know

• 1Auto Loader (cloudFiles source) is the recommended method for incremental, scalable ingestion of new files landing in cloud storage, using directory listing or file notification mode to detect new files.
• 2COPY INTO is a simpler, idempotent SQL command best suited for periodically ingesting new files into a Delta table when file volumes are moderate and full Auto Loader complexity isn't needed.
• 3Supported structured file types include Parquet, Delta, ORC, and Avro (with built-in schema), while semi-structured/text types like JSON and CSV require explicit or inferred schemas and are more prone to schema drift.
• 4Schema inference and evolution settings (cloudFiles.schemaEvolutionMode, mergeSchema) must be configured deliberately, since new columns or type mismatches can break pipelines if not anticipated.
• 5Batch (full) extraction reprocesses entire datasets and is simplest but costly at scale, whereas incremental extraction (using checkpoints, watermarks, or Auto Loader's file tracking) processes only new or changed data for efficiency.
• 6File format choice impacts performance: columnar formats (Parquet, Delta) support predicate pushdown and compression, while row-based formats (CSV, JSON) are easier to produce but slower to query and larger in size.

2 · Check your understanding