Streaming logs into Log Analytics in Azure Monitor

Azure Databricks clusters can stream Spark driver/executor logs, listener events, and metrics into a Log Analytics workspace in near real time, enabling centralized monitoring and KQL-based analysis. This is achieved by attaching the Azure Databricks monitoring library and a cluster-scoped init script that forwards logs to Azure Monitor, separate from workspace-level diagnostic (audit) logging.

1 · Learn the must-know

• 1Log streaming requires manually attaching the Azure Databricks monitoring (spark-listener) library JAR and an init script to each cluster; it is not enabled by default.
• 2The init script and Spark config must reference the target Log Analytics workspace ID and shared key (often stored as cluster environment variables or secrets) so the cluster can authenticate to Azure Monitor.
• 3Streamed data lands in custom log tables such as SparkListenerEvent_CL, SparkLoggingEvent_CL, and SparkMetric_CL, which you query using Kusto Query Language (KQL) in the Log Analytics workspace.
• 4This cluster-level log streaming is distinct from workspace diagnostic settings, which send platform/audit logs (e.g., notebook, job, cluster events) to Log Analytics, Event Hubs, or a storage account.
• 5Because it is applied per cluster, the library and init script must be configured on every cluster (or via cluster policies/global init scripts) that needs its Spark logs and metrics streamed.
• 6Log Analytics ingestion and retention incur cost based on data volume, so teams should filter or sample verbose Spark logs to control expenses.

2 · Check your understanding