Turning on Photon, picking a Databricks runtime, and enabling machine learning on compute

When creating an Azure Databricks compute resource (cluster), you configure key feature settings on the compute creation/edit page, including the Databricks Runtime version, whether to enable Photon acceleration, and whether the runtime includes Machine Learning libraries. These choices affect performance, cost, available APIs, and pre-installed libraries for your workloads.

1 · Learn the must-know

• 1Databricks Runtime versions bundle a specific Apache Spark version plus optimizations, and Databricks recommends using the latest LTS (Long Term Support) runtime for production workloads for stability and extended support.
• 2Photon is a native vectorized query engine that accelerates SQL and DataFrame workloads on Delta/Parquet data; it is enabled via a toggle at compute creation and requires a Photon-compatible runtime, and it incurs a higher DBU rate than non-Photon compute.
• 3Selecting a 'Machine Learning' runtime variant (e.g., ML-flavored Databricks Runtime) pre-installs common ML/DL libraries (such as scikit-learn, TensorFlow, PyTorch, XGBoost, and MLflow) so you don't need to manually install them via libraries or init scripts.
• 4ML runtimes and Photon are configured independently, but not all combinations are supported, so the ML runtime selection may limit Photon availability depending on the version chosen.
• 5Runtime and Photon settings can only be set at cluster creation/edit time and require a cluster restart to take effect, they cannot be changed on a running cluster without a restart.
• 6Choosing an unsupported or deprecated runtime version can cause compatibility issues with newer workspace features or libraries, so runtime selection should align with workload requirements (e.g., standard vs. ML vs. GPU-enabled runtimes).

2 · Check your understanding