Selecting the right engine for a real-time workload

Microsoft Fabric offers multiple ways to process streaming data, and choosing the right engine depends on your latency, transformation complexity, and destination needs. The main options are Eventstream for no-code ingestion/routing, Spark Structured Streaming for complex custom transformations, and KQL (Eventhouse) for high-speed analytical querying of streaming data.

1 · Learn the must-know

• 1Eventstream is the low-code/no-code engine for ingesting, filtering, and routing streaming data from sources like Event Hubs, IoT Hub, or Kafka to destinations such as Lakehouse, Eventhouse, or KQL Database.
• 2Spark Structured Streaming (via Fabric notebooks or jobs) is the right choice when you need complex, custom transformation logic, windowed aggregations, or joins with batch data using PySpark/Scala/SQL.
• 3KQL (Kusto Query Language) against an Eventhouse/KQL Database is optimized for near-real-time analytical queries on high-volume streaming data with sub-second latency, ideal for dashboards and anomaly detection.
• 4Data Activator is not a transformation engine but a rule-based trigger service that monitors streaming data (often from Eventstream or KQL) and fires actions/alerts when conditions are met.
• 5A common gotcha: Eventstream handles simple transformations (filter, aggregate, expand) natively, but for advanced stateful processing you must route data to Spark or a custom endpoint rather than trying to force it into Eventstream alone.
• 6Choice often depends on latency requirements: Eventstream/KQL suit sub-second to seconds-level near-real-time needs, while Spark Structured Streaming is better for micro-batch scenarios (seconds to minutes) requiring heavier compute or ML integration.

2 · Check your understanding