Azure Data Lake Storage Gen2

Azure Data Lake Storage Gen2 (also known as ADLS Gen2) is a next-generation data lake solution for big data analytics. Azure Data Lake Storage Gen2 builds Azure Data Lake Storage Gen1 capabilities—file system semantics, file-level security, and scale—into Azure Blob storage, with its low-cost tiered storage, high availability, and disaster recovery features. The Azure Blob File System (ABFS) driver provides the interface to ADLS Gen2 storage. The ABFS driver, included in the Databricks Runtime, supports standard file system semantics on Azure Blob storage.

The articles in this section describe methods for accessing ADLS Gen2 storage and frequently encountered problems:

• To learn how to use storage account access keys to quickly access ADLS Gen2 storage from Azure Databricks for testing, development, or experimentation, see Get started with Azure Data Lake Storage Gen2.

• For production and multi-user scenarios, there are several authentication methods to access ADLS Gen2 storage from Azure Databricks. Select a method according to your requirements:

  • To provide workspace users access to ADLS Gen2 storage based on each user’s permissions, see Access Azure Data Lake Storage using Azure Active Directory credential passthrough.
  • To provide multiple workspace users access to a common set of folders or files, see Access Azure Data Lake Storage Gen2 using OAuth 2.0 with an Azure service principal.
  • To have fine-grained control over access to storage resources, see Access Azure Data Lake Storage Gen2 directly using a SAS token provider to define custom access policies.

• For frequently asked questions and known issues, see Azure Data Lake Storage Gen2 frequently asked questions and known issues.