Ingesting with Spark Structured Streaming

Spark Structured Streaming lets you treat data arriving in cloud storage, message buses, or Delta tables as an unbounded table, processing new records incrementally as they arrive. On Azure Databricks it is commonly used with Auto Loader and Delta Lake to build reliable, scalable ingestion pipelines with exactly-once processing guarantees.

1 · Learn the must-know

• 1Auto Loader (cloudFiles source) incrementally and efficiently detects new files landing in cloud storage without listing the entire directory each time, using file notification or directory listing modes.
• 2Every streaming write requires a checkpointLocation, which stores offsets and state so the stream can resume exactly where it left off after a failure or restart.
• 3Structured Streaming supports two trigger models: micro-batch processing (default, or with a specified interval) and continuous/low-latency processing, plus Trigger.AvailableNow for running a stream once over all currently available data like a batch job.
• 4readStream and writeStream DataFrame APIs look similar to batch APIs, but streaming DataFrames are unbounded and only support a subset of transformations (no operations requiring full-dataset sorting without watermarking, for example).
• 5Writing a stream to a Delta table (outputMode append or complete, or using foreachBatch for upserts via merge) is the standard sink pattern and enables downstream consumers to read the table as either batch or stream.
• 6Schema inference and evolution can be enabled with Auto Loader (cloudFiles.schemaLocation, cloudFiles.schemaEvolutionMode) so new columns in source files are handled without manually redefining the schema each time.

2 · Check your understanding