Configuring Azure Monitor alerts

Azure Databricks integrates with Azure Monitor to surface diagnostic logs and metrics from your workspace, clusters, and jobs, enabling you to configure alert rules that notify you when defined thresholds or conditions are met. Alerts are built on top of diagnostic settings that route Databricks logs into a Log Analytics workspace, where you write Kusto (KQL) queries to define the alert logic. This lets data engineers proactively monitor pipeline health, job failures, and resource utilization rather than reacting after the fact.

1 · Learn the must-know

• 1You must first enable diagnostic logging on the Databricks workspace and send logs (e.g., cluster events, job run events, notebook activity) to a Log Analytics workspace before Azure Monitor can alert on them.
• 2Alert rules in Azure Monitor are created from a Log Analytics query (KQL) that scans the ingested Databricks logs; the alert fires when the query results meet a defined threshold over a chosen evaluation frequency and time window.
• 3Action groups define what happens when an alert fires (email, SMS, webhook, Logic App, Azure Function, ITSM integration, etc.) and are attached to the alert rule separately from the alert logic itself.
• 4There is inherent latency between an event occurring in Databricks and the log appearing in Log Analytics, so near-real-time alerting has a delay and very tight evaluation frequencies may not be meaningful.
• 5Metric-based alerts can also be configured on cluster-level resource metrics (CPU, memory, etc.) surfaced through Azure Monitor for the underlying VM scale sets, separate from log-based query alerts.
• 6A common exam gotcha is confusing Databricks-native job notifications (email/webhook alerts on job success/failure configured within a job's settings) with Azure Monitor alerts, which are broader, infrastructure/log-based, and configured outside Databricks in the Azure portal.

2 · Check your understanding