Bringing semi-structured and unstructured data into governed Delta tables

Lakeflow Connect provides managed, no-code/low-code ingestion connectors that land data from SaaS applications, databases, and files directly into Unity Catalog-governed Delta tables, handling schema evolution and incremental refresh automatically. For semi-structured formats like JSON, Databricks stores nested structs, arrays, and maps natively in Delta's variant/struct columns, avoiding manual flattening at ingest time. Governance (access control, lineage, auditing) is enforced by Unity Catalog on the resulting Delta tables regardless of source connector.

1 · Learn the must-know

• 1Lakeflow Connect ingestion pipelines write directly to Unity Catalog-managed Delta tables, so catalog, schema, and table permissions apply immediately without a separate registration step.
• 2JSON and other semi-structured data can be ingested with schema inference and evolution enabled, letting nested fields (structs, arrays, maps) be preserved as native Delta types rather than flattened strings.
• 3Auto Loader (cloudFiles) is the underlying incremental file-ingestion mechanism used for landing JSON, CSV, Parquet, and other files from cloud object storage into Delta tables, using checkpointing to track processed files exactly-once.
• 4Unstructured or binary data (images, PDFs, etc.) is typically ingested using the binaryFile format, which stores file content plus metadata (path, modification time, length) as columns in a Delta table for downstream processing.
• 5Managed connectors reduce the need for custom ETL code but still require defining a target catalog/schema and appropriate Unity Catalog permissions (USE CATALOG, USE SCHEMA, CREATE TABLE) before a pipeline can land data.
• 6Schema drift in nested JSON (new or missing fields) is handled via schema evolution settings; ignoring this can cause pipeline failures or silently dropped columns if not configured correctly.

2 · Check your understanding