Creating and configuring a job

Azure Databricks Jobs let you orchestrate one or more tasks (notebooks, Python scripts, JARs, SQL queries, dbt, or pipelines) as an automated workflow with its own compute, schedule, and monitoring. Jobs are created and managed via the Workflows UI, the Jobs API, or the Databricks CLI, and support multi-task DAGs with dependencies.

1 · Learn the must-know

• 1A job consists of one or more tasks arranged in a DAG, where you set task dependencies (predecessors) to control execution order.
• 2Each task can run on a dedicated new job cluster, an existing all-purpose cluster, or a serverless compute option, and different tasks in the same job can use different clusters.
• 3Jobs can be triggered manually, on a cron-based schedule, continuously, or via file arrival/table update triggers, and support concurrent run limits.
• 4You can parameterize jobs and tasks using job parameters or task values, and pass parameters into notebooks via widgets.
• 5Configure retries, timeouts, and alerting (email or webhook notifications) at the task or job level to handle failures and SLA breaches.
• 6Job run history, task-level logs, and output are all visible in the Workflows UI for monitoring and troubleshooting, and permissions can be set to control who can view, manage, or run the job.

2 · Check your understanding