Handling a continuous flow of records with Spark's structured streaming

Spark Structured Streaming in Microsoft Fabric enables near-real-time processing of data within notebooks using DataFrames, treating a live data stream as an unbounded table that is incrementally processed. It integrates with Fabric's lakehouse architecture, allowing you to read from sources like files or event streams and write results to Delta tables using familiar Spark APIs.

1 · Learn the must-know

• 1Structured Streaming uses the same DataFrame/Dataset API as batch processing, so most transformations (select, filter, groupBy, joins) work identically on streaming DataFrames.
• 2You create a streaming DataFrame with spark.readStream (e.g., specifying format like 'cloudFiles' for Autoloader-style ingestion or 'delta' for reading a Delta table as a stream) and write output with df.writeStream.
• 3Output modes matter: 'append' only writes new rows and is required for most non-aggregated queries, 'complete' rewrites the full result table (used with aggregations), and 'update' writes only changed rows.
• 4Checkpointing (via the checkpointLocation option) is mandatory for fault tolerance: it tracks progress and offsets so a restarted stream resumes exactly where it left off without data loss or duplication.
• 5Trigger intervals control processing cadence: default is micro-batch as fast as possible, but you can set a fixed interval (e.g., trigger(processingTime='1 minute')) or use Trigger.AvailableNow/once for batch-like one-time processing of available data.
• 6In Fabric, streaming writes commonly target Delta tables in the Lakehouse, enabling downstream consumption via SQL endpoints, Power BI, or further Spark jobs; watermarking is used to handle late-arriving data in stateful aggregations.

2 · Check your understanding