Use PowerShell to manage directories, files, and ACLs in Azure Data Lake Storage Gen2 (preview)

This article shows you how to use PowerShell to create and manage directories, files, and permissions in storage accounts that has hierarchical namespace (HNS) enabled.

Important

The PowerShell module that is featured in this article is currently in public preview.

Gen1 to Gen2 mapping | Give feedback

Prerequisites

  • An Azure subscription. See Get Azure free trial.
  • A storage account that has hierarchical namespace (HNS) enabled. Follow these instructions to create one.
  • .NET Framework is 4.7.2 or greater installed. See Download .NET Framework.
  • PowerShell version 5.1 or higher.

Install PowerShell modules

  1. Verify that the version of PowerShell that have installed is 5.1 or higher by using the following command.

    echo $PSVersionTable.PSVersion.ToString() 
    

    To upgrade your version of PowerShell, see Upgrading existing Windows PowerShell

  2. Install the latest PowershellGet module. Then, close and reopen the Powershell console.

    install-Module PowerShellGet –Repository PSGallery –Force 
    
  3. Install Az.Storage preview module.

    install-Module Az.Storage -Repository PSGallery -RequiredVersion 1.9.1-preview –AllowPrerelease –AllowClobber –Force 
    

    For more information about how to install PowerShell modules, see Install the Azure PowerShell module

Connect to the account

Open a Windows PowerShell command window, and then sign in to your Azure subscription with the Connect-AzAccount command and follow the on-screen directions.

Connect-AzAccount

If your identity is associated with more than one subscription, then set your active subscription to subscription of the storage account that you want create and manage directories in. In this example, replace the <subscription-id> placeholder value with the ID of your subscription.

Select-AzSubscription -SubscriptionId <subscription-id>

Next, choose how you want your commands to obtain authorization to the storage account.

Option 1: Obtain authorization by using Azure Active Directory (AD)

With this approach, the system ensures that your user account has the appropriate role-based access control (RBAC) assignments and ACL permissions.

$ctx = New-AzStorageContext -StorageAccountName '<storage-account-name>' -UseConnectedAccount

Option 2: Obtain authorization by using the storage account key

With this approach, the system doesn't check RBAC or ACL permissions.

$storageAccount = Get-AzStorageAccount -ResourceGroupName "<resource-group-name>" -AccountName "<storage-account-name>"
$ctx = $storageAccount.Context

Create a file system

A file system acts as a container for your files. You can create one by using the New-AzDatalakeGen2FileSystem cmdlet.

This example creates a file system named my-file-system.

$filesystemName = "my-file-system"
New-AzDatalakeGen2FileSystem -Context $ctx -Name $filesystemName

Create a directory

Create a directory reference by using the New-AzDataLakeGen2Item cmdlet.

This example adds a directory named my-directory to a file system.

$filesystemName = "my-file-system"
$dirname = "my-directory/"
New-AzDataLakeGen2Item -Context $ctx -FileSystem $filesystemName -Path $dirname -Directory

This example adds the same directory, but also sets the permissions, umask, property values, and metadata values.

$dir = New-AzDataLakeGen2Item -Context $ctx -FileSystem $filesystemName -Path $dirname -Directory -Permission rwxrwxrwx -Umask ---rwx---  -Property @{"ContentEncoding" = "UDF8"; "CacheControl" = "READ"} -Metadata  @{"tag1" = "value1"; "tag2" = "value2" }

Show directory properties

This example gets a directory by using the Get-AzDataLakeGen2Item cmdlet, and then prints property values to the console.

$filesystemName = "my-file-system"
$dirname = "my-directory/"
$dir =  Get-AzDataLakeGen2Item -Context $ctx -FileSystem $filesystemName -Path $dirname
$dir.ACL
$dir.Permissions
$dir.Directory.PathProperties.Group
$dir.Directory.PathProperties.Owner
$dir.Directory.Metadata
$dir.Directory.Properties

Rename or move a directory

Rename or move a directory by using the Move-AzDataLakeGen2Item cmdlet.

This example renames a directory from the name my-directory to the name my-new-directory.

$filesystemName = "my-file-system"
$dirname = "my-directory/"
$dirname2 = "my-new-directory/"
Move-AzDataLakeGen2Item -Context $ctx -FileSystem $filesystemName -Path $dirname -DestFileSystem $filesystemName -DestPath $dirname2

This example moves a directory named my-directory to a subdirectory of my-directory-2 named my-subdirectory. This example also applies a umask to the subdirectory.

$filesystemName = "my-file-system"
$dirname = "my-directory/"
$dirname2 = "my-directory-2/my-subdirectory/"
Move-AzDataLakeGen2Item -Context $ctx -FileSystem $filesystemName -Path $dirname1 -DestFileSystem $filesystemName -DestPath $dirname2 -Umask --------x -PathRenameMode Posix

Delete a directory

Delete a directory by using the Remove-AzDataLakeGen2Item cmdlet.

This example deletes a directory named my-directory.

$filesystemName = "my-file-system"
$dirname = "my-directory/"
Remove-AzDataLakeGen2Item  -Context $ctx -FileSystem $filesystemName -Path $dirname 

You can use the -Force parameter to remove the file without a prompt.

Download from a directory

Download a file from a directory by using the Get-AzDataLakeGen2ItemContent cmdlet.

This example downloads a file named upload.txt from a directory named my-directory.

$filesystemName = "my-file-system"
$filePath = "my-directory/upload.txt"
$downloadFilePath = "download.txt"
Get-AzDataLakeGen2ItemContent -Context $ctx -FileSystem $filesystemName -Path $filePath -Destination $downloadFilePath

List directory contents

List the contents of a directory by using the Get-AzDataLakeGen2ChildItem cmdlet.

This example lists the contents of a directory named my-directory.

$filesystemName = "my-file-system"
$dirname = "my-directory/"
Get-AzDataLakeGen2ChildItem -Context $ctx -FileSystem $filesystemName -Path $dirname

This example doesn't return values for the ACL, Permissions, Group, and Owner properties. To obtain those values, use the -FetchPermission parameter.

The following example lists the contents of the same directory, but it also uses the -FetchPermission parameter to return values for the ACL, Permissions, Group, and Owner properties.

$filesystemName = "my-file-system"
$dirname = "my-directory/"
$properties = Get-AzDataLakeGen2ChildItem -Context $ctx -FileSystem $filesystemName -Path $dirname -Recurse -FetchPermission
$properties.ACL
$properties.Permissions
$properties.Group
$properties.Owner

To list the contents of a file system, omit the -Path parameter from the command.

Upload a file to a directory

Upload a file to a directory by using the New-AzDataLakeGen2Item cmdlet.

This example uploads a file named upload.txt to a directory named my-directory.

$localSrcFile =  "upload.txt"
$filesystemName = "my-file-system"
$dirname = "my-directory/"
$destPath = $dirname + (Get-Item $localSrcFile).Name
New-AzDataLakeGen2Item -Context $ctx -FileSystem $filesystemName -Path $destPath -Source $localSrcFile -Force 

This example uploads the same file, but then sets the permissions, umask, property values, and metadata values of the destination file. This example also prints these values to the console.

$file = New-AzDataLakeGen2Item -Context $ctx -FileSystem $filesystemName -Path $destPath -Source $localSrcFile -Permission rwxrwxrwx -Umask ---rwx--- -Property @{"ContentEncoding" = "UDF8"; "CacheControl" = "READ"} -Metadata  @{"tag1" = "value1"; "tag2" = "value2" }
$file1
$file1.File.Metadata
$file1.File.Properties

Show file properties

This example gets a file by using the Get-AzDataLakeGen2Item cmdlet, and then prints property values to the console.

$filepath =  "my-directory/upload.txt"
$filesystemName = "my-file-system"
$file = Get-AzDataLakeGen2Item -Context $ctx -FileSystem $filesystemName -Path $filepath
$file
$file.ACL
$file.Permissions
$file.File.PathProperties.Group
$file.File.PathProperties.Owner
$file.File.Metadata
$file.File.Properties

Delete a file

Delete a file by using the Remove-AzDataLakeGen2Item cmdlet.

This example deletes a file named upload.txt.

$filesystemName = "my-file-system"
$filepath = "upload.txt"
Remove-AzDataLakeGen2Item  -Context $ctx -FileSystem $filesystemName -Path $filepath 

You can use the -Force parameter to remove the file without a prompt.

Manage access permissions

You can get, set, and update access permissions of directories and files.

Note

If you're using Azure Active Directory (Azure AD) to authorize commands, then make sure that your security principal has been assigned the Storage Blob Data Owner role. To learn more about how ACL permissions are applied and the effects of changing them, see Access control in Azure Data Lake Storage Gen2.

Get directory and file permissions

Get the ACL of a directory or file by using the Get-AzDataLakeGen2Itemcmdlet.

This example gets the ACL of a directory, and then prints the ACL to the console.

$filesystemName = "my-file-system"
$dirname = "my-directory/"
$dir = Get-AzDataLakeGen2Item -Context $ctx -FileSystem $filesystemName -Path $dirname
$dir.ACL

This example gets the ACL of a file and then prints the ACL to the console.

$filePath = "my-directory/upload.txt"
$file = Get-AzDataLakeGen2Item -Context $ctx -FileSystem $filesystemName -Path $filePath
$file.ACL

The following image shows the output after getting the ACL of a directory.

Get ACL output

In this example, the owning user has read, write, and execute permissions. The owning group has only read and execute permissions. For more information about access control lists, see Access control in Azure Data Lake Storage Gen2.

Set directory and file permissions

Use the New-AzDataLakeGen2ItemAclObject cmdlet to create an ACL for the owning user, owning group, or other users. Then, use the Update-AzDataLakeGen2Item cmdlet to commit the ACL.

This example sets the ACL on a directory for the owning user, owning group, or other users, and then prints the ACL to the console.

$filesystemName = "my-file-system"
$dirname = "my-directory/"
$acl = New-AzDataLakeGen2ItemAclObject -AccessControlType user -Permission rw- 
$acl = New-AzDataLakeGen2ItemAclObject -AccessControlType group -Permission rw- -InputObject $acl 
$acl = New-AzDataLakeGen2ItemAclObject -AccessControlType other -Permission "-wx" -InputObject $acl
Update-AzDataLakeGen2Item -Context $ctx -FileSystem $filesystemName -Path $dirname -Acl $acl
$dir = Get-AzDataLakeGen2Item -Context $ctx -FileSystem $filesystemName -Path $dirname
$dir.ACL

This example sets the ACL on a file for the owning user, owning group, or other users, and then prints the ACL to the console.

$filesystemName = "my-file-system"
$filePath = "my-directory/upload.txt"
$acl = New-AzDataLakeGen2ItemAclObject -AccessControlType user -Permission rw- 
$acl = New-AzDataLakeGen2ItemAclObject -AccessControlType group -Permission rw- -InputObject $acl 
$acl = New-AzDataLakeGen2ItemAclObject -AccessControlType other -Permission "-wx" -InputObject $acl
Update-AzDataLakeGen2Item -Context $ctx -FileSystem $filesystemName -Path $filePath -Acl $acl
$file = Get-AzDataLakeGen2Item -Context $ctx -FileSystem $filesystemName -Path $filePath
$file.ACL

The following image shows the output after setting the ACL of a file.

Get ACL output

In this example, the owning user and owning group have only read and write permissions. All other users have write and execute permissions. For more information about access control lists, see Access control in Azure Data Lake Storage Gen2.

Update directory and file permissions

Use the Get-AzDataLakeGen2Item cmdlet to get the ACL of a directory or file. Then, use the New-AzDataLakeGen2ItemAclObject cmdlet to create a new ACL entry. Use the Update-AzDataLakeGen2Item cmdlet to apply the new ACL.

This example gives a group write and execute permission on a directory.

$filesystemName = "my-file-system"
$dirname = "my-directory/"
$Id = "xxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"

# Get the directory ACL
$acl = (Get-AzDataLakeGen2Item -Context $ctx -FileSystem $filesystemName -Path $dirname).ACL

# Create the new ACL object.
[Collections.Generic.List[System.Object]]$aclnew =$acl

# To avoid duplicate ACL, remove the ACL entries that will be added later.
foreach ($a in $aclnew)
{
    if ($a.AccessControlType -eq "group" -and $a.DefaultScope -eq $true-and $a.EntityId -eq $id)
    {
        $aclnew.Remove($a);
        break;
    }
}

# Add ACL Entries
$aclnew = New-AzDataLakeGen2ItemAclObject -AccessControlType group -EntityId $id -Permission "-wx" -DefaultScope -InputObject $aclnew

# Update ACL on server
Update-AzDataLakeGen2Item -Context $ctx -FileSystem $filesystemName -Path $dirname -Acl $aclnew  

This example gives a group write and execute permission on a file.

$filesystemName = "my-file-system"
$fileName = "my-directory/upload.txt"
$Id = "xxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"

# Get the file ACL
$acl = (Get-AzDataLakeGen2Item -Context $ctx -FileSystem $filesystemName -Path $fileName).ACL

# Create the new ACL object.
[Collections.Generic.List[System.Object]]$aclnew =$acl

# To avoid duplicate ACL, remove the ACL entries that will be added later.
foreach ($a in $aclnew)
{
    if ($a.AccessControlType -eq "group" -and $a.DefaultScope -eq $true-and $a.EntityId -eq $id)
    {
        $aclnew.Remove($a);
        break;
    }
}

# Add ACL Entries
$aclnew = New-AzDataLakeGen2ItemAclObject -AccessControlType group -EntityId $id -Permission "-wx" -DefaultScope -InputObject $aclnew

# Update ACL on server
Update-AzDataLakeGen2Item -Context $ctx -FileSystem $filesystemName -Path $fileName -Acl $aclnew 

Set permissions on all items in a file system

You can use the Get-AzDataLakeGen2Item and the -Recurse parameter together with the Update-AzDataLakeGen2Item cmdlet to recursively to set the ACL of all directories and files in a file system.

$filesystemName = "my-file-system"
$acl = New-AzDataLakeGen2ItemAclObject -AccessControlType user -Permission rw- 
$acl = New-AzDataLakeGen2ItemAclObject -AccessControlType group -Permission rw- -InputObject $acl 
$acl = New-AzDataLakeGen2ItemAclObject -AccessControlType other -Permission "-wx" -InputObject $acl
Get-AzDataLakeGen2ChildItem -Context $ctx -FileSystem $filesystemName -Recurse -FetchPermission | Update-AzDataLakeGen2Item -Acl $acl

Gen1 to Gen2 Mapping

The following table shows how the cmdlets used for Data Lake Storage Gen1 map to the cmdlets for Data Lake Storage Gen2.

Data Lake Storage Gen1 cmdlet Data Lake Storage Gen2 cmdlet Notes
Get-AzDataLakeStoreChildItem Get-AzDataLakeGen2ChildItem By default, the Get-AzDataLakeGen2ChildItem cmdlet only lists the first level child items. The -Recurse parameter lists child items recursively.
Get-AzDataLakeStoreItem
Get-AzDataLakeStoreItemAclEntry
Get-AzDataLakeStoreItemOwner
Get-AzDataLakeStoreItemPermission
Get-AzDataLakeGen2Item The output items of the Get-AzDataLakeGen2Item cmdlet has these properties: Acl, Owner, Group, Permission.
Get-AzDataLakeStoreItemContent Get-AzDataLakeGen2FileContent The Get-AzDataLakeGen2FileContent cmdlet download file content to local file.
Move-AzDataLakeStoreItem Move-AzDataLakeGen2Item
New-AzDataLakeStoreItem New-AzDataLakeGen2Item This cmdlet uploads the new file content from a local file.
Remove-AzDataLakeStoreItem Remove-AzDataLakeGen2Item
Set-AzDataLakeStoreItemOwner
Set-AzDataLakeStoreItemPermission
Set-AzDataLakeStoreItemAcl
Update-AzDataLakeGen2Item The Update-AzDataLakeGen2Item cmdlet updates a single item only, and not recursively. If want to update recursively, list items by using the Get-AzDataLakeStoreChildItem cmdlet, then pipeline to the Update-AzDataLakeGen2Item cmdlet.
Test-AzDataLakeStoreItem Get-AzDataLakeGen2Item The Get-AzDataLakeGen2Item cmdlet will report an error if the item doesn't exist.

See also