December 2018

  • Article

These features and Azure Databricks platform improvements were released in December 2018.

Note

Releases are staged. Your Azure Databricks account may not be updated until up to a week after the initial release date.

Databricks Runtime 5.1 for Machine Learning (Beta) release

Dec 18, 2018

Databricks Runtime 5.1 ML is built on top of Databricks Runtime 5.1. For information on what’s new in Databricks Runtime 5.1, see the Databricks Runtime 5.1 (unsupported) release notes. In addition to the updates to existing libraries in Databricks Runtime 5.0 ML (unsupported), Databricks Runtime 5.1 ML includes the following new features:

  • PyTorch for building deep learning networks.

See the complete release notes for Databricks Runtime 5.1 ML (unsupported).

Databricks Runtime 5.1 release

December 18, 2018

Databricks Runtime 5.1 is now available. Databricks Runtime 5.1 includes Apache Spark 2.4.0, new Delta Lake and Structured Streaming features and upgrades, and upgraded Python, R, and Java and Scala libraries. For details, see Databricks Runtime 5.1 (unsupported).

Library UI

The library UI update originally released in version 2.85 was reverted on December 7, 2018.

Access Azure Data Lake Storage Gen1 automatically with your Microsoft Entra ID credentials

December 5-11, 2018: Version 2.86

You can now authenticate automatically to Azure Data Lake Storage Gen1 from Azure Databricks clusters using the same Microsoft Entra ID identity that you use to log into Azure Databricks.

Simply enable your cluster for Microsoft Entra ID credential passthrough, and commands that you run on that cluster will be able to read and write your data in Azure Data Lake Storage Gen1 without requiring you to configure service principal credentials for access to storage.

Important

The ability to access Azure Data Lake Storage Gen1 using your Microsoft Entra ID credentials is in Public Preview. It requires Databricks Runtime 5.1, which is currently in Beta but will soon be generally available.

For more information, see Access automatically with your Microsoft Entra ID (formerly Azure Active Directory) credentials.