Ingesting streams from Azure Event Hubs

Azure Databricks ingests streaming data from Azure Event Hubs using Spark Structured Streaming, either via the native Event Hubs connector or the Kafka-compatible endpoint since Event Hubs implements the Kafka protocol. The connection is configured through a readStream source with connection details (namespace, entity path, and credentials) supplied as options, and the resulting DataFrame is processed like any other streaming source.

1 · Learn the must-know

• 1Use spark.readStream.format("eventhubs") with the azure-eventhubs-spark connector, or format("kafka") pointing at the Event Hubs Kafka endpoint (port 9093) since Event Hubs is Kafka-protocol compatible.
• 2Event Hubs connection strings must be encrypted with EventHubsUtils.encrypt() before being placed in the EventHubsConf, and should be retrieved from a Databricks secret scope (backed by Azure Key Vault or Databricks-managed) rather than hardcoded.
• 3A checkpoint location (typically on DBFS or ADLS) is required for the streaming query to track consumed offsets and enable fault-tolerant, exactly-once processing on restart.
• 4The number of Event Hubs partitions caps the maximum parallelism of the Structured Streaming job, so partition count should be sized to the desired throughput.
• 5Use maxEventsPerTrigger (or maxOffsetsPerTrigger for the Kafka connector) to control micro-batch size and prevent overwhelming the cluster with backlog data.
• 6Consumer groups in Event Hubs should be dedicated per Databricks streaming application to avoid throttling or offset conflicts with other consumers reading the same event hub.

2 · Check your understanding