Configuring automatic restarts for jobs and pipelines

Azure Databricks Jobs support automatic retries so transient failures in tasks or job runs don't require manual intervention. You configure retry behavior at the task level (or job level for scheduled jobs) by specifying the number of retry attempts and an optional delay between retries. This helps ensure pipeline reliability without building custom retry logic.

1 · Learn the must-know

• 1Retries are configured per task in the Jobs UI or via the Jobs API using the 'max_retries' and 'min_retry_interval_millis' properties.
• 2Setting 'max_retries' to -1 enables unlimited retries, which should be used cautiously since it can mask persistent failures and increase compute costs.
• 3You can enable 'retry_on_timeout' so that a task automatically retries if it fails due to exceeding its configured timeout.
• 4Delta Live Tables (DLT) pipelines have built-in automatic retry and recovery logic for pipeline runs and updates, distinct from Jobs task retry settings.
• 5Retries apply to task failures, not to successful tasks that produce incorrect results, so retry logic won't fix logic or data quality errors.
• 6For multi-task jobs, you can configure retries independently per task, allowing critical tasks to have more retry attempts than less critical ones.

2 · Check your understanding