These features and Azure Databricks platform improvements were released in December 2018.
Releases are staged. Your Azure Databricks account may not be updated until up to a week after the initial release date.
Databricks Runtime 5.1 for Machine Learning (Beta) release
Dec 18, 2018
Databricks Runtime 5.1 ML is built on top of Databricks Runtime 5.1. For information on what’s new in Databricks Runtime 5.1, see the Databricks Runtime 5.1 (Unsupported) release notes. In addition to the updates to existing libraries in Databricks Runtime 5.0 ML (Beta), Databricks Runtime 5.1 ML includes the following new features:
- PyTorch for building deep learning networks.
See the complete release notes for Databricks Runtime 5.1 ML (Beta).
Databricks Runtime 5.1 release
December 18, 2018
Databricks Runtime 5.1 is now available. Databricks Runtime 5.1 includes Apache Spark 2.4.0, new Delta Lake and Structured Streaming features and upgrades, and upgraded Python, R, and Java and Scala libraries. For details, see Databricks Runtime 5.1 (Unsupported).
The library UI update originally released in version 2.85 was reverted on December 7, 2018.
Access Azure Data Lake Storage Gen1 automatically with your Azure AD credentials
December 5-11, 2018: Version 2.86
You can now authenticate automatically to Azure Data Lake Storage Gen1 from Azure Databricks clusters using the same Azure Active Directory (Azure AD) identity that you use to log into Azure Databricks.
Simply enable your cluster for Azure AD credential passthrough, and commands that you run on that cluster will be able to read and write your data in Azure Data Lake Storage Gen1 without requiring you to configure service principal credentials for access to storage.
The ability to access Azure Data Lake Storage Gen1 using your Azure AD credentials is in Public Preview. It requires Databricks Runtime 5.1, which is currently in Beta but will soon be generally available.
For more information, see Access automatically with your Azure Active Directory credentials.