Setting up Lakeflow Connect to ingest from enterprise sources reliably
Lakeflow Connect provides managed, native connectors for ingesting data from enterprise sources (SaaS applications, databases via CDC, file systems) directly into Unity Catalog-governed Delta tables, eliminating the need for custom ingestion pipelines. It handles schema evolution, incremental sync, and governance automatically, so learners must know which source types are supported and how ingestion pipelines integrate with Unity Catalog's permission model.
Must-know
- Lakeflow Connect ingestion pipelines write directly into Unity Catalog-managed Delta tables, so target catalogs, schemas, and tables must exist within a Unity Catalog metastore before configuring ingestion.
- Connectors support incremental ingestion patterns, including change data capture (CDC) from databases, so downstream tables reflect ongoing source changes without full reloads.
- Ingestion pipelines run as governed workloads, meaning access to both source connections and destination tables is controlled through Unity Catalog permissions and, for source credentials, through securable connection objects.
- Schema evolution (new columns or type changes at the source) is handled automatically by the connector, but structural changes should still be validated against downstream consumers before promotion.
- Configuring a connection requires source-specific authentication (e.g., OAuth, API tokens, or database credentials) stored securely and referenced by the pipeline rather than hardcoded.
- Ingested tables inherit lineage and audit tracking within Unity Catalog, enabling data stewards to trace table origin back to the source connector and pipeline run.
A data engineer is setting up a Lakeflow Connect ingestion pipeline to bring data from a Salesforce source into a Unity Catalog schema. The source table has a reliable 'LastModifiedDate' field, and the engineer wants new and changed records captured automatically without manually re-ingesting the full dataset each run. Which ingestion approach should they configure?
What you have tried across Databricks DEA's objectives, not a readiness score.
Databricks Intelligence Platform
Data Ingestion and Loading
- Batch, streaming, and incremental loading patterns, and where the data comes from
- Loading files from cloud storage into governed tables with COPY INTO
- Landing data with Auto Loader, and handling schema enforcement and evolution
- Setting up Lakeflow Connect to ingest from enterprise sources reliably
- Pulling data through JDBC, ODBC, or REST clients and scheduling the job
- Choosing the right ingestion method for a given volume, frequency, and governance need
- Bringing semi-structured and unstructured data into governed Delta tables
Data Transformation and Modeling
- Cleaning bronze data into silver tables with PySpark and SQL
- Joining and combining DataFrames with the different join and union types
- Reshaping columns, rows, and arrays in a table
- Deduplicating and aggregating DataFrames
- Tuning Spark's core parameters and measuring what changed
- Building Gold-layer views and tables for BI and analytics
- Validating Silver and Gold datasets for quality
Working with Lakeflow Jobs
Implementing CI/CD
- Branching, committing, and opening pull requests from inside the Databricks workspace
- Promoting one codebase across dev, test, and prod with bundle variables and overrides
- Packaging and deploying jobs and pipelines with Automation Bundles
- Validating and managing bundle deployments from the Databricks CLI
Troubleshooting, Monitoring, and Optimization
- Spotting performance trends in a job's run history
- Reading job status, task graphs, and failure rates to monitor pipeline health
- Diagnosing skew, shuffle, and spill from Spark UI stage metrics
- What Liquid Clustering and predictive optimization actually do
- Diagnosing cluster startup failures, library conflicts, and out-of-memory errors
Governance and Security
Coverage checked against the published exam guide on Jul 25, 2026.
These are independent practice questions, written against this certification's published exam guide. They are not the certification vendor's own questions, and not the real exam.