Azure Databricks provides an environment that makes it easy to build, train, and deploy deep learning models at scale. Many deep learning libraries are available in Databricks Runtime ML, a machine learning runtime that provides a ready-to-go environment for machine learning and data science. For deep learning libraries not included in Databricks Runtime ML, you can either install libraries as an Azure Databricks library or use init scripts to install libraries on clusters upon creation.
Graphics processing units (GPUs) can accelerate deep learning tasks. For information about creating GPU-enabled Azure Databricks clusters, see GPU-enabled clusters. Databricks Runtime ML includes installed GPU hardware drivers and NVIDIA libraries such as CUDA.
A typical deep learning workflow involves the phases data preparation, training, and inference. This section gives guidelines on deep learning in Azure Databricks.
- Data preparation
- Single node training
- Distributed training
- Model inference
- Reference solutions