Using pipeline expectations to manage data quality

Lakeflow Spark Declarative Pipelines let you enforce data quality rules directly on streaming tables and materialized views using expectations, which are boolean constraints evaluated as each batch of records is processed. Expectations control what happens to rows that fail validation—keep, drop, or halt the pipeline—and their pass/fail metrics are captured automatically for monitoring.

1 · Learn the must-know

• 1In Python, expectations are declared with the @dlt.expect(), @dlt.expect_or_drop(), and @dlt.expect_or_fail() decorators, each taking a constraint name and a SQL boolean expression.
• 2In SQL, expectations use the CONSTRAINT name EXPECT (condition) syntax, optionally followed by ON VIOLATION DROP ROW or ON VIOLATION FAIL UPDATE to change the default warn behaviour.
• 3The default violation action is 'warn': failing rows are still written to the target table but counted as failures in pipeline metrics, so a passive expectation alone will not stop bad data from landing.
• 4Multiple related constraints can be grouped and applied together using expect_all, expect_all_or_drop, and expect_all_or_fail, which take a Python dictionary mapping constraint names to expressions.
• 5Expectation pass/fail counts per constraint are recorded in the pipeline's event log and visible in the pipeline UI graph, enabling data quality monitoring over time without extra instrumentation code.
• 6Expectations only apply to records processed by the decorated streaming table or materialized view definition within the pipeline; they do not retroactively validate data already in the table or apply to plain batch/notebook code outside the pipeline.

2 · Check your understanding