Ingesting batch and streaming data with Lakeflow Connect

Lakeflow Connect provides managed, no-code/low-code connectors for ingesting data from SaaS applications, databases, and files directly into Databricks as Delta tables, simplifying pipeline setup for both batch and streaming ingestion. It integrates with Delta Live Tables (DLT) pipelines and Unity Catalog to handle schema evolution, incremental ingestion, and governance automatically. It reduces the need for custom connector code by offering prebuilt, managed connectors maintained by Databricks.

1 · Learn the must-know

• 1Lakeflow Connect uses managed connectors (e.g., for Salesforce, SharePoint, ServiceNow, Workday, and databases like SQL Server) that handle authentication, incremental extraction, and schema drift automatically.
• 2Ingested data lands in Unity Catalog-governed Delta tables, enabling immediate lineage, access control, and auditing without extra configuration.
• 3Batch connectors typically use scheduled or triggered full/incremental refreshes, while streaming-capable sources (like database CDC) can continuously ingest changes with low latency using Delta Live Tables' streaming tables.
• 4Lakeflow Connect pipelines are built and managed via DLT under the hood, so ingestion logic benefits from DLT's built-in data quality expectations and pipeline monitoring/observability UI.
• 5A common exam gotcha: not all sources support true streaming ingestion—some connectors only support scheduled batch pulls, so you must check per-connector capabilities before assuming CDC/streaming support.
• 6Setup typically requires configuring a connection (credentials/OAuth) once, then reusing it across multiple ingestion pipelines, reducing duplicated credential management for the same source system.

2 · Check your understanding