Bringing semi-structured and unstructured data into governed Delta tables
Lakeflow Connect provides managed, no-code/low-code ingestion connectors that land data from SaaS applications, databases, and files directly into Unity Catalog-governed Delta tables, handling schema evolution and incremental refresh automatically. For semi-structured formats like JSON, Databricks stores nested structs, arrays, and maps natively in Delta's variant/struct columns, avoiding manual flattening at ingest time. Governance (access control, lineage, auditing) is enforced by Unity Catalog on the resulting Delta tables regardless of source connector.
Must-know
- Lakeflow Connect ingestion pipelines write directly to Unity Catalog-managed Delta tables, so catalog, schema, and table permissions apply immediately without a separate registration step.
- JSON and other semi-structured data can be ingested with schema inference and evolution enabled, letting nested fields (structs, arrays, maps) be preserved as native Delta types rather than flattened strings.
- Auto Loader (cloudFiles) is the underlying incremental file-ingestion mechanism used for landing JSON, CSV, Parquet, and other files from cloud object storage into Delta tables, using checkpointing to track processed files exactly-once.
- Unstructured or binary data (images, PDFs, etc.) is typically ingested using the binaryFile format, which stores file content plus metadata (path, modification time, length) as columns in a Delta table for downstream processing.
- Managed connectors reduce the need for custom ETL code but still require defining a target catalog/schema and appropriate Unity Catalog permissions (USE CATALOG, USE SCHEMA, CREATE TABLE) before a pipeline can land data.
- Schema drift in nested JSON (new or missing fields) is handled via schema evolution settings; ignoring this can cause pipeline failures or silently dropped columns if not configured correctly.
A data engineering team needs to land nested JSON files from a partner SFTP-style cloud storage location into a Unity Catalog table, automatically evolve the schema as new fields appear, and avoid reprocessing files already loaded on each run. Which ingestion approach meets these requirements?
What you have tried across Databricks DEA's objectives, not a readiness score.
Databricks Intelligence Platform
Data Ingestion and Loading
- Batch, streaming, and incremental loading patterns, and where the data comes from
- Loading files from cloud storage into governed tables with COPY INTO
- Landing data with Auto Loader, and handling schema enforcement and evolution
- Setting up Lakeflow Connect to ingest from enterprise sources reliably
- Pulling data through JDBC, ODBC, or REST clients and scheduling the job
- Choosing the right ingestion method for a given volume, frequency, and governance need
- Bringing semi-structured and unstructured data into governed Delta tables
Data Transformation and Modeling
- Cleaning bronze data into silver tables with PySpark and SQL
- Joining and combining DataFrames with the different join and union types
- Reshaping columns, rows, and arrays in a table
- Deduplicating and aggregating DataFrames
- Tuning Spark's core parameters and measuring what changed
- Building Gold-layer views and tables for BI and analytics
- Validating Silver and Gold datasets for quality
Working with Lakeflow Jobs
Implementing CI/CD
- Branching, committing, and opening pull requests from inside the Databricks workspace
- Promoting one codebase across dev, test, and prod with bundle variables and overrides
- Packaging and deploying jobs and pipelines with Automation Bundles
- Validating and managing bundle deployments from the Databricks CLI
Troubleshooting, Monitoring, and Optimization
- Spotting performance trends in a job's run history
- Reading job status, task graphs, and failure rates to monitor pipeline health
- Diagnosing skew, shuffle, and spill from Spark UI stage metrics
- What Liquid Clustering and predictive optimization actually do
- Diagnosing cluster startup failures, library conflicts, and out-of-memory errors
Governance and Security
Coverage checked against the published exam guide on Jul 25, 2026.
These are independent practice questions, written against this certification's published exam guide. They are not the certification vendor's own questions, and not the real exam.