Ordering the steps in a data pipeline

In Databricks Workflows, pipeline order of operations is controlled by defining tasks and explicit dependencies (predecessors) that form a Directed Acyclic Graph (DAG), ensuring each task runs only after its required upstream tasks succeed. Correctly sequencing tasks—such as ingestion before transformation before serving (medallion architecture: bronze → silver → gold)—prevents data quality issues and wasted compute from processing incomplete or stale data.

1 · Learn the must-know

• 1A Databricks Job can contain multiple tasks, and dependencies between tasks are set explicitly via 'Depends on', creating a DAG rather than a strictly linear sequence.
• 2Tasks with no dependency relationship (no shared predecessor path) can run in parallel automatically, so pipeline design should identify which steps are truly independent to optimize runtime.
• 3The medallion architecture pattern (Bronze for raw ingestion, Silver for cleaned/conformed data, Gold for aggregated/business-level data) dictates a natural order of operations that pipeline task dependencies should enforce.
• 4Task run conditions (e.g., 'All succeeded', 'At least one succeeded', 'All done') determine whether downstream tasks execute based on the outcome of upstream tasks, affecting pipeline flow control.
• 5If a task fails and downstream tasks depend on it, those downstream tasks are skipped by default, so proper ordering and error-handling logic (e.g., using conditional task run-if settings) is critical to avoid silent pipeline gaps.
• 6Delta Live Tables (DLT) pipelines automatically infer execution order from table/view dependencies declared in code, so in DLT the order of operations is determined by data lineage rather than manual task sequencing.

2 · Check your understanding