Ingesting batch and streaming data from notebooks

Azure Databricks notebooks can ingest data in both batch and streaming modes using Apache Spark APIs, letting you read from cloud storage, files, or streaming sources into DataFrames for processing. Auto Loader (cloudFiles) and Structured Streaming are the primary tools for scalable, incremental ingestion directly from notebooks, while spark.read is used for one-time batch loads.

1 · Learn the must-know

• 1Auto Loader (format('cloudFiles')) incrementally and efficiently processes new files as they arrive in cloud storage, tracking state to avoid reprocessing and supporting schema inference/evolution.
• 2Batch ingestion typically uses spark.read.format(...).load(path) for a static, one-time read of files (CSV, JSON, Parquet, Delta, etc.) into a DataFrame.
• 3Streaming ingestion uses spark.readStream with a defined or inferred schema, and writeStream with a checkpointLocation to enable fault-tolerant, exactly-once processing.
• 4Streaming queries require a trigger (e.g., processingTime, availableNow) and an output mode (append, complete, update) appropriate to the transformation being performed.
• 5COPY INTO is a simpler, idempotent SQL-based alternative to Auto Loader for batch-loading new files into a Delta table, tracking already-loaded files automatically.
• 6Notebooks can mix batch and streaming logic in the same session, but streaming cells run continuously until stopped, canceled, or the cluster terminates, so care is needed when scheduling jobs that include streaming writes.

2 · Check your understanding