Pulling data through JDBC, ODBC, or REST clients and scheduling the job

External systems (databases, APIs, SaaS apps) can be pulled into Databricks by running JDBC/ODBC or REST client code inside a notebook, writing results as files to cloud storage or directly as Delta tables in Unity Catalog. These notebooks are then wrapped in Lakeflow Jobs so the ingestion runs on a schedule or trigger without manual intervention.

1 · Learn the must-know

• 1Notebooks can use standard Python/Scala JDBC drivers (e.g. via Spark's spark.read.jdbc/format('jdbc')) to connect to relational sources, or REST/HTTP client libraries (e.g. requests) to pull data from APIs.
• 2Spark's JDBC reader supports partitioning options (partitionColumn, lowerBound, upperBound, numPartitions) to parallelize large table pulls and avoid single-connection bottlenecks.
• 3Data landed via REST calls typically arrives as JSON/text in driver memory or a local file first, then must be converted to a Spark DataFrame and written out (e.g. as Delta) rather than assumed to be distributed automatically.
• 4Writing directly into Unity Catalog–governed tables requires the cluster or SQL warehouse to have a Unity Catalog-enabled access mode, and the identity running the job needs appropriate USE CATALOG/USE SCHEMA/CREATE TABLE grants.
• 5Credentials for JDBC connections or API tokens should be stored using Databricks secrets (secret scopes) rather than hardcoded in notebook cells.
• 6Lakeflow Jobs orchestrate these ingestion notebooks with schedules, triggers, retries, and alerting, and can chain them as tasks alongside downstream transformation steps.

2 · Check your understanding