Configuring job triggers

Azure Databricks Jobs can be started automatically using Scheduled, Continuous, or File arrival triggers, in addition to manual runs or API/CLI invocation. Choosing the right trigger type and understanding its concurrency and latency behavior is essential for reliable pipeline orchestration.

1 · Learn the must-know

• 1Scheduled triggers use Quartz cron syntax and require you to select a time zone, and the schedule can be paused without deleting the job definition.
• 2Continuous triggers automatically restart the job as soon as the previous run finishes or fails, and only support a single concurrent run at a time.
• 3File arrival triggers watch an external cloud storage location (e.g., via a Unity Catalog external location) and start a run when new files are detected, but detection is periodic (polling), not instantaneous, so there is inherent latency.
• 4The 'Maximum concurrent runs' setting controls whether multiple runs of the same job can execute simultaneously, which matters when triggers fire faster than jobs complete.
• 5Jobs can also be triggered externally via the REST API or CLI (e.g., 'runs submit'/'run now'), which is useful for orchestration from external schedulers or CI/CD pipelines.
• 6A job can only have one primary trigger type configured at a time (schedule, continuous, or file arrival), though manual/API-triggered runs remain possible regardless of the configured trigger.

2 · Check your understanding