Batch, streaming, and incremental loading patterns, and where the data comes from

Databricks supports three core ingestion patterns: one-time batch loads, continuous/triggered streaming with Structured Streaming, and incremental batch loads that pick up only new files since the last run. Local/cloud file ingestion is typically handled via Auto Loader (cloudFiles) or COPY INTO, while Lakeflow Connect provides standard connectors (for common SaaS/file sources) and managed connectors (fully hosted ingestion pipelines) for scalable, low-maintenance data import.

1 · Learn the must-know

• 1Auto Loader (cloudFiles format) incrementally and idempotently processes new files landing in cloud storage, tracking processed files via checkpointing so files are not reprocessed.
• 2COPY INTO is a SQL command best suited for idempotent, incremental batch loading of files into a Delta table when file counts are moderate (thousands, not millions); Auto Loader scales better for very large or continuously arriving file volumes.
• 3Streaming ingestion in Databricks uses Structured Streaming with triggers: trigger(availableNow=True) processes all currently available data once then stops (micro-batch style incremental processing), while continuous/fixed-interval triggers keep the stream running.
• 4Lakeflow Connect standard connectors provide simplified, UI/config-driven ingestion from common external sources (e.g., cloud storage, message buses) without hand-written streaming code.
• 5Lakeflow Connect managed connectors are fully managed, Databricks-hosted ingestion pipelines for enterprise applications and databases (e.g., Salesforce, SQL Server, Workday) that handle scheduling, schema evolution, and incremental refresh automatically.
• 6Schema inference and evolution (rescued data column) are key gotchas with Auto Loader: unexpected or new columns are captured in a rescued data column rather than causing job failure, unless schema enforcement is explicitly configured.

2 · Check your understanding