Designing task logic for Lakeflow Jobs

Lakeflow Jobs (formerly Databricks Jobs) let you orchestrate multi-step workflows by chaining tasks of different types with defined dependencies, conditional logic, and parameter passing. Designing task logic means choosing the right task type, wiring dependencies and run conditions, and passing values between tasks to build reliable, reusable pipelines.

1 · Learn the must-know

• 1A job can combine multiple task types in one DAG, including Notebook, Python script, Python wheel, SQL, JAR, dbt, Run Job, Pipeline (DLT), and If/else condition tasks.
• 2Task dependencies are set via 'Depends on', and each dependent task's 'Run if dependencies' setting controls execution based on upstream success, failure, or completion (all succeeded, at least one succeeded, all done, at least one failed, all failed).
• 3The For Each task lets a single task definition iterate over a list of inputs concurrently, with configurable concurrency limits, avoiding the need to duplicate tasks manually.
• 4dbutils.jobs.taskValues.set() and .get() (task values) pass small data outputs between tasks in a DAG, while job/task parameters pass static or dynamic values (including dynamic value references like {{job.start_time}}) into tasks at runtime.
• 5Each task can have its own cluster (job cluster or existing all-purpose cluster), retry policy (max retries, retry interval), timeout, and email/webhook notifications, allowing independent tuning per task rather than for the whole job.
• 6The Run Job task type lets one job trigger another existing job as a task, enabling modular, reusable pipeline components instead of duplicating logic across jobs.

2 · Check your understanding