Get started with Azure Data Lake Storage Gen1 using Azure PowerShell
On Feb 29, 2024 Azure Data Lake Storage Gen1 will be retired. For more information, see the official announcement. If you use Azure Data Lake Storage Gen1, make sure to migrate to Azure Data Lake Storage Gen2 prior to that date. To learn how, see Migrate Azure Data Lake Storage from Gen1 to Gen2
Learn how to use Azure PowerShell to create an Azure Data Lake Storage Gen1 account and perform basic operations such as create folders, upload and download data files, delete your account, etc. For more information about Data Lake Storage Gen1, see Overview of Data Lake Storage Gen1.
This article has been updated to use the Azure Az PowerShell module. The Az PowerShell module is the recommended PowerShell module for interacting with Azure. To get started with the Az PowerShell module, see Install Azure PowerShell. To learn how to migrate to the Az PowerShell module, see Migrate Azure PowerShell from AzureRM to Az.
- An Azure subscription. See Get Azure free trial.
- Azure PowerShell 1.0 or greater. See How to install and configure Azure PowerShell.
This article uses a simpler authentication approach with Data Lake Storage Gen1 where you're prompted to enter your Azure account credentials. The access level to Data Lake Storage Gen1 account and file system is then governed by the access level of the logged in user. However, there are other approaches to authenticate with Data Lake Storage Gen1, which are end-user authentication or service-to-service authentication. For instructions and more information on how to authenticate, see End-user authentication or Service-to-service authentication.
Create a Data Lake Storage Gen1 account
From your desktop, open a new Windows PowerShell window. Enter the following snippet to log in to your Azure account, set the subscription, and register the Data Lake Storage Gen1 provider. When prompted to log in, make sure you log in as one of the subscription administrators/owner:
# Log in to your Azure account Connect-AzAccount # List all the subscriptions associated to your account Get-AzSubscription # Select a subscription Set-AzContext -SubscriptionId <subscription ID> # Register for Azure Data Lake Storage Gen1 Register-AzResourceProvider -ProviderNamespace "Microsoft.DataLakeStore"
A Data Lake Storage Gen1 account is associated with an Azure resource group. Start by creating a resource group.
$resourceGroupName = "<your new resource group name>" New-AzResourceGroup -Name $resourceGroupName -Location "East US 2"
Create a Data Lake Storage Gen1 account. The name you specify must only contain lowercase letters and numbers.
$dataLakeStorageGen1Name = "<your new Data Lake Storage Gen1 account name>" New-AzDataLakeStoreAccount -ResourceGroupName $resourceGroupName -Name $dataLakeStorageGen1Name -Location "East US 2"
Verify that the account is successfully created.
Test-AzDataLakeStoreAccount -Name $dataLakeStorageGen1Name
The output for the cmdlet should be True.
Create directory structures
You can create directories under your Data Lake Storage Gen1 account to manage and store data.
Specify a root directory.
$myrootdir = "/"
Create a new directory called mynewdirectory under the specified root.
New-AzDataLakeStoreItem -Folder -AccountName $dataLakeStorageGen1Name -Path $myrootdir/mynewdirectory
Verify that the new directory is successfully created.
Get-AzDataLakeStoreChildItem -AccountName $dataLakeStorageGen1Name -Path $myrootdir
It should show an output as shown in the following screenshot:
You can upload your data to Data Lake Storage Gen1 directly at the root level, or to a directory that you created within the account. The snippets in this section demonstrate how to upload some sample data to the directory (mynewdirectory) you created in the previous section.
If you are looking for some sample data to upload, you can get the Ambulance Data folder from the Azure Data Lake Git Repository. Download the file and store it in a local directory on your computer, such as C:\sampledata.
Import-AzDataLakeStoreItem -AccountName $dataLakeStorageGen1Name ` -Path "C:\sampledata\vehicle1_09142014.csv" ` -Destination $myrootdir\mynewdirectory\vehicle1_09142014.csv
Rename, download, and delete data
To rename a file, use the following command:
Move-AzDataLakeStoreItem -AccountName $dataLakeStorageGen1Name ` -Path $myrootdir\mynewdirectory\vehicle1_09142014.csv ` -Destination $myrootdir\mynewdirectory\vehicle1_09142014_Copy.csv
To download a file, use the following command:
Export-AzDataLakeStoreItem -AccountName $dataLakeStorageGen1Name ` -Path $myrootdir\mynewdirectory\vehicle1_09142014_Copy.csv ` -Destination "C:\sampledata\vehicle1_09142014_Copy.csv"
To delete a file, use the following command:
Remove-AzDataLakeStoreItem -AccountName $dataLakeStorageGen1Name ` -Paths $myrootdir\mynewdirectory\vehicle1_09142014_Copy.csv
When prompted, enter Y to delete the item. If you have more than one file to delete, you can provide all the paths separated by comma.
Remove-AzDataLakeStoreItem -AccountName $dataLakeStorageGen1Name ` -Paths $myrootdir\mynewdirectory\vehicle1_09142014.csv, $myrootdir\mynewdirectoryvehicle1_09142014_Copy.csv
Delete your account
Use the following command to delete your Data Lake Storage Gen1 account.
Remove-AzDataLakeStoreAccount -Name $dataLakeStorageGen1Name
When prompted, enter Y to delete the account.