Reading job status, task graphs, and failure rates to monitor pipeline health
Lakeflow Jobs (formerly Databricks Workflows) provides a UI to monitor multi-task job runs through a DAG-based task graph, letting you see run status, spot failed or blocked upstream tasks, and review run history and durations. Understanding the job run states and how task dependencies propagate failures is key to diagnosing pipeline health quickly.
Must-know
- Job run states include Pending, Running, Succeeded, Failed, and Terminated/Skipped; a task's color in the DAG view (e.g., red for failed, gray for skipped) instantly shows where a pipeline broke.
- If an upstream task fails, downstream tasks that depend on it are skipped by default unless the task's dependency condition (e.g., run even if a dependency fails) is explicitly configured.
- The Runs tab lists historical runs with start time, duration, and status, enabling you to track run times and failure rates over time to spot trends or regressions.
- Clicking a task node in the DAG opens task-level details including logs, output, and duration, which is the fastest way to isolate the root cause of a failure in a multi-task job.
- Job-level and task-level retry policies can be configured; retried attempts appear in the run history, so a run can show 'Succeeded' overall even after earlier task attempts failed.
- Email or webhook notifications can be configured on job start, success, or failure, allowing proactive monitoring without manually checking the Jobs UI.
A data engineering team opens the Lakeflow Jobs run graph for a nightly pipeline. The task 'load_orders' shows a red failed icon, while its two downstream tasks 'transform_orders' and 'load_summary' both show gray, never-run icons. A separate unrelated branch in the same job completed successfully. What does the DAG indicate?
What you have tried across Databricks DEA's objectives, not a readiness score.
Databricks Intelligence Platform
Data Ingestion and Loading
- Batch, streaming, and incremental loading patterns, and where the data comes from
- Loading files from cloud storage into governed tables with COPY INTO
- Landing data with Auto Loader, and handling schema enforcement and evolution
- Setting up Lakeflow Connect to ingest from enterprise sources reliably
- Pulling data through JDBC, ODBC, or REST clients and scheduling the job
- Choosing the right ingestion method for a given volume, frequency, and governance need
- Bringing semi-structured and unstructured data into governed Delta tables
Data Transformation and Modeling
- Cleaning bronze data into silver tables with PySpark and SQL
- Joining and combining DataFrames with the different join and union types
- Reshaping columns, rows, and arrays in a table
- Deduplicating and aggregating DataFrames
- Tuning Spark's core parameters and measuring what changed
- Building Gold-layer views and tables for BI and analytics
- Validating Silver and Gold datasets for quality
Working with Lakeflow Jobs
Implementing CI/CD
- Branching, committing, and opening pull requests from inside the Databricks workspace
- Promoting one codebase across dev, test, and prod with bundle variables and overrides
- Packaging and deploying jobs and pipelines with Automation Bundles
- Validating and managing bundle deployments from the Databricks CLI
Troubleshooting, Monitoring, and Optimization
- Spotting performance trends in a job's run history
- Reading job status, task graphs, and failure rates to monitor pipeline health
- Diagnosing skew, shuffle, and spill from Spark UI stage metrics
- What Liquid Clustering and predictive optimization actually do
- Diagnosing cluster startup failures, library conflicts, and out-of-memory errors
Governance and Security
Coverage checked against the published exam guide on Jul 27, 2026.
These are independent practice questions, written against this certification's published exam guide. They are not the certification vendor's own questions, and not the real exam.