Installing libraries on a compute resource

Libraries add third-party or custom code (Python, JAR, R, etc.) to a cluster so notebooks and jobs can import them. Libraries can be installed at the cluster (compute) level or scoped to a notebook session, and can come from PyPI, Maven, CRAN, workspace files, Unity Catalog volumes, or uploaded files.

1 · Learn the must-know

• 1Cluster-installed libraries are available to all notebooks attached to that cluster and are reinstalled automatically on cluster restart.
• 2Notebook-scoped libraries (installed via %pip or %conda magic commands) apply only to the current notebook session and do not persist across cluster restarts or affect other users on shared clusters.
• 3Library sources include PyPI, Maven Central, CRAN, workspace files, Unity Catalog volumes, and DBFS/direct file upload, selectable from the cluster's Libraries tab.
• 4Installing a library on a running cluster may require the cluster to restart or the library to be reinstalled to take effect on all nodes, especially for JAR/init-script based libraries.
• 5Library conflicts (version mismatches between cluster-installed and notebook-scoped libraries) can cause import errors, so notebook-scoped %pip installs are recommended for isolating dependencies per workload.
• 6On job clusters, libraries can be specified as part of the job/task definition so the correct dependencies are installed automatically each time the ephemeral cluster starts.

2 · Check your understanding