Wiring notebook, SQL, dashboard, and pipeline tasks into a job's task graph

Lakeflow Jobs orchestrate multi-task workflows as a DAG, where each node is a task (notebook, SQL query, dashboard refresh, pipeline, Python script, etc.) and edges define dependencies via 'Depends on'. Tasks run on job clusters or serverless compute, and a task only fires once all its upstream dependencies finish successfully, enabling fan-out/fan-in patterns and conditional branching.

1 · Learn the must-know

• 1A task's 'Depends on' field can list multiple upstream tasks, and by default all of them must succeed for the downstream task to run.
• 2Task run conditions can be changed from 'All succeeded' to options like 'At least one succeeded', 'All done', or 'At least one failed' to control branching on failure.
• 3Notebook, SQL, dashboard, and pipeline tasks each require a compute target: notebook and SQL script tasks can run on job/all-purpose clusters or serverless, SQL query/dashboard tasks run against a SQL warehouse, and pipeline tasks trigger a Lakeflow Declarative Pipelines update rather than running on a job cluster.
• 4A dashboard task refreshes a Lakeview dashboard and depends on the SQL warehouse and underlying queries being current, so it's commonly chained after notebook/pipeline tasks that update the source tables.
• 5Parameters and values can be passed between tasks using task values (dbutils.jobs.taskValues) or job/task parameters referenced with {{}} syntax, letting downstream tasks consume upstream outputs.
• 6The DAG structure is visualized in the Jobs UI graph view, and a job run fails overall if any task without a satisfied alternate condition fails, though 'Repair run' lets you rerun only the failed and downstream tasks.

2 · Check your understanding