Choosing batch or streaming as the loading method

Azure Databricks supports both batch and streaming data loading, and choosing the right approach depends on latency requirements, data source characteristics, and how often new data arrives. Batch loading processes data in discrete chunks (e.g., using Apache Spark DataFrames or COPY INTO), while streaming uses Structured Streaming or Auto Loader for continuous, incremental ingestion. Auto Loader is the recommended method for incrementally and efficiently ingesting new files as they arrive in cloud storage.

1 · Learn the must-know

• 1Auto Loader (cloudFiles format) incrementally and efficiently processes new data files as they arrive in cloud storage, using checkpointing to track processed files and avoid reprocessing.
• 2COPY INTO is a SQL command best suited for idempotent, scheduled batch loading of new files from a stage location into a Delta table, and is efficient for thousands of files but less efficient at scale compared to Auto Loader for millions of files.
• 3Structured Streaming provides a unified API for both batch and streaming workloads, allowing the same code to process data incrementally as it arrives or as a one-time batch job (using Trigger.AvailableNow or Trigger.Once).
• 4Delta Lake's ACID transactions and schema enforcement/evolution features apply consistently to both batch and streaming writes, ensuring data reliability regardless of loading method.
• 5Streaming loads require a checkpoint location to maintain state and ensure exactly-once processing semantics, while batch loads typically rely on table versioning or partition overwrites for consistency.
• 6Choosing batch is appropriate for large, infrequent loads (e.g., daily/hourly ETL) with predictable schemas, whereas streaming suits near-real-time use cases with continuously arriving data from sources like Kafka, Event Hubs, or cloud storage.

2 · Check your understanding