Known issues with Azure Data Lake Storage Gen2

This article describes limitations and known issues for accounts that have the hierarchical namespace feature enabled.

Note

Some of the features described in this article might not be supported in accounts that have Network File System (NFS) 3.0 support enabled. To view a table that shows the impact of feature support when various capabilities are enabled, see Blob Storage feature support in Azure Storage accounts.

Supported Blob storage features

An increasing number of Blob storage features now work with accounts that have a hierarchical namespace. For a complete list, see Blob Storage features available in Azure Data Lake Storage Gen2.

Supported Azure service integrations

Azure Data Lake Storage Gen2 supports several Azure services that you can use to ingest data, perform analytics, and create visual representations. For a list of supported Azure services, see Azure services that support Azure Data Lake Storage Gen2.

For more information, see Azure services that support Azure Data Lake Storage Gen2.

Supported open source platforms

Several open source platforms support Data Lake Storage Gen2. For a complete list, see Open source platforms that support Azure Data Lake Storage Gen2.

For more information, see Open source platforms that support Azure Data Lake Storage Gen2.

Blob storage APIs

Data Lake Storage Gen2 APIs, NFS 3.0, and Blob APIs can operate on the same data.

This section describes issues and limitations with using blob APIs, NFS 3.0, and Data Lake Storage Gen2 APIs to operate on the same data.

  • You cannot use blob APIs, NFS 3.0, and Data Lake Storage APIs to write to the same instance of a file. If you write to a file by using Data Lake Storage Gen2 or APIs or NFS 3.0, then that file's blocks won't be visible to calls to the Get Block List blob API. The only exception is when using you are overwriting. You can overwrite a file/blob using either API or with NFS 3.0 by using the zero-truncate option.

  • When you use the List Blobs operation without specifying a delimiter, the results will include both directories and blobs. If you choose to use a delimiter, use only a forward slash (/). This is the only supported delimiter.

  • If you use the Delete Blob API to delete a directory, that directory will be deleted only if it's empty. This means that you can't use the Blob API delete directories recursively.

These Blob REST APIs aren't supported:

Unmanaged VM disks are not supported in accounts that have a hierarchical namespace. If you want to enable a hierarchical namespace on a storage account, place unmanaged VM disks into a storage account that doesn't have the hierarchical namespace feature enabled.

Support for setting access control lists (ACLs) recursively

The ability to apply ACL changes recursively from parent directory to child items is generally available. In the current release of this capability, you can apply ACL changes by using Azure Storage Explorer, PowerShell, Azure CLI, and the .NET, Java, and Python SDK. Support is not yet available for the Azure portal.

Access control lists (ACL) and anonymous read access

If anonymous read access has been granted to a container, then ACLs have no effect on that container or the files in that container.

AzCopy

Use only the latest version of AzCopy (AzCopy v10). Earlier versions of AzCopy such as AzCopy v8.1, are not supported.

Azure Storage Explorer

Use only versions 1.6.0 or higher.

Storage Explorer in the Azure portal

ACLs are not yet supported.

Third party applications

Third party applications that use REST APIs to work will continue to work if you use them with Data Lake Storage Gen2. Applications that call Blob APIs will likely work.

Storage Analytics logs (classic)

The setting for retention days is not yet supported, but you can delete logs manually by using any supported tool such as Azure Storage Explorer, REST or an SDK.

Lifecycle management policies with premium tier for Azure Data Lake Storage

You can't move data that's stored in the premium tier between hot, cool, and archive tiers. However, you can copy data from the premium tier to the hot access tier in a different account.

Windows Azure Storage Blob (WASB) driver (unsupported with Data Lake Storage Gen2)

Currently, the WASB driver, which was designed to work with the Blob API only, encounters problems in a few common scenarios. Specifically, when it is a client to a hierarchical namespace-enabled storage account. Multi-protocol access on Data Lake Storage won't mitigate these issues.

For the time being (and most likely the foreseeable future), we won't support customers using the WASB driver as a client to a hierarchical namespace-enabled storage account. Instead, we recommend that you opt to use the Azure Blob File System (ABFS) driver in your Hadoop environment. If you are trying to migrate off of an on-premise Hadoop environment with a version earlier than Hadoop branch-3, then please open an Azure Support ticket so that we can get in touch with you on the right path forward for you and your organization.

Soft delete for blobs capability (currently in preview)

If parent directories for soft-deleted files or directories are renamed, the soft-deleted items may not be displayed correctly in the Azure portal. In such cases you can use PowerShell or Azure CLI to list and restore the soft-deleted items.

Events

If your account has an event subscription, read operations on the secondary endpoint will result in an error. To resolve this issue, remove event subscriptions.

Tip

Read access to the secondary endpoint is available only when you enable read-access geo-redundant storage (RA-GRS) or read-access geo-zone-redundant storage (RA-GZRS).