Building a pipeline with Lakeflow Spark Declarative Pipelines

Lakeflow Spark Declarative Pipelines (formerly Delta Live Tables/DLT) let you define ETL logic declaratively using SQL or PySpark, and the pipeline engine handles orchestration, dependency resolution, cluster management, and execution. You define datasets as streaming tables or materialized views and Databricks automatically infers the DAG, manages incremental processing, and tracks lineage.

1 · Learn the must-know

• 1Pipelines are defined using SQL CREATE STREAMING TABLE / CREATE MATERIALIZED VIEW syntax or the equivalent @dlt.table / @dlt.view decorators in PySpark, with the pipeline engine inferring the dependency DAG from query references rather than requiring manual orchestration code.
• 2A pipeline is created and configured (source notebooks/files, target catalog and schema, compute settings, and Triggered vs Continuous mode) either through the Databricks UI Pipelines editor, Databricks Asset Bundles, or the REST API/CLI, not by directly running notebooks interactively.
• 3Triggered pipelines run once and stop (suitable for scheduled batch-like execution via a Databricks Job), while Continuous pipelines keep running to process new data with low latency; choosing the right mode is a common exam distinction.
• 4Expectations (constraints defined with EXPECT/EXPECT OR DROP/EXPECT OR FAIL in SQL, or expect_all/expect_all_or_drop/expect_all_or_fail decorators in Python) enforce data quality and are evaluated per pipeline run, with violation metrics surfaced in the pipeline event log.
• 5The pipeline's target catalog and schema settings determine where streaming tables and materialized views are published as Unity Catalog objects, so incorrect target configuration is a common source of failures or misplaced tables.
• 6Pipeline execution details, data quality metrics, and lineage are all visible in the pipeline's event log and graph view in the Pipelines UI, which is the primary tool for monitoring and troubleshooting pipeline runs.

2 · Check your understanding