Get started with Azure Data Lake Store using Azure PowerShell

Learn how to use Azure PowerShell to create an Azure Data Lake Store account and perform basic operations such as create folders, upload and download data files, delete your account, etc. For more information about Data Lake Store, see Overview of Data Lake Store.

Prerequisites

Authentication

This article uses a simpler authentication approach with Data Lake Store where you are prompted to enter your Azure account credentials. The access level to Data Lake Store account and file system is then governed by the access level of the logged in user. However, there are other approaches as well to authenticate with Data Lake Store, which are end-user authentication or service-to-service authentication. For instructions and more information on how to authenticate, see End-user authentication or Service-to-service authentication.

Create an Azure Data Lake Store account

  1. From your desktop, open a new Windows PowerShell window. Enter the following snippet to log in to your Azure account, set the subscription, and register the Data Lake Store provider. When prompted to log in, make sure you log in as one of the subscription admininistrators/owner:

     # Log in to your Azure account
     Connect-AzureRmAccount
    
     # List all the subscriptions associated to your account
     Get-AzureRmSubscription
    
     # Select a subscription
     Set-AzureRmContext -SubscriptionId <subscription ID>
    
     # Register for Azure Data Lake Store
     Register-AzureRmResourceProvider -ProviderNamespace "Microsoft.DataLakeStore"
    
  2. An Azure Data Lake Store account is associated with an Azure Resource Group. Start by creating an Azure Resource Group.

     $resourceGroupName = "<your new resource group name>"
     New-AzureRmResourceGroup -Name $resourceGroupName -Location "East US 2"
    

    Create an Azure Resource Group

  3. Create an Azure Data Lake Store account. The name you specify must only contain lowercase letters and numbers.

     $dataLakeStoreName = "<your new Data Lake Store name>"
     New-AzureRmDataLakeStoreAccount -ResourceGroupName $resourceGroupName -Name $dataLakeStoreName -Location "East US 2"
    

    Create an Azure Data Lake Store account

  4. Verify that the account is successfully created.

     Test-AzureRmDataLakeStoreAccount -Name $dataLakeStoreName
    

    The output for the cmdlet should be True.

Create directory structures in your Azure Data Lake Store

You can create directories under your Azure Data Lake Store account to manage and store data.

  1. Specify a root directory.

     $myrootdir = "/"
    
  2. Create a new directory called mynewdirectory under the specified root.

     New-AzureRmDataLakeStoreItem -Folder -AccountName $dataLakeStoreName -Path $myrootdir/mynewdirectory
    
  3. Verify that the new directory is successfully created.

     Get-AzureRmDataLakeStoreChildItem -AccountName $dataLakeStoreName -Path $myrootdir
    

    It should show an output as shown in the following screenshot:

    Verify Directory

Upload data to your Azure Data Lake Store

You can upload your data to Data Lake Store directly at the root level, or to a directory that you created within the account. The snippets in this section demonstrate how to upload some sample data to the directory (mynewdirectory) you created in the previous section.

If you are looking for some sample data to upload, you can get the Ambulance Data folder from the Azure Data Lake Git Repository. Download the file and store it in a local directory on your computer, such as C:\sampledata.

Import-AzureRmDataLakeStoreItem -AccountName $dataLakeStoreName -Path "C:\sampledata\vehicle1_09142014.csv" -Destination $myrootdir\mynewdirectory\vehicle1_09142014.csv

Rename, download, and delete data from your Data Lake Store

To rename a file, use the following command:

Move-AzureRmDataLakeStoreItem -AccountName $dataLakeStoreName -Path $myrootdir\mynewdirectory\vehicle1_09142014.csv -Destination $myrootdir\mynewdirectory\vehicle1_09142014_Copy.csv

To download a file, use the following command:

Export-AzureRmDataLakeStoreItem -AccountName $dataLakeStoreName -Path $myrootdir\mynewdirectory\vehicle1_09142014_Copy.csv -Destination "C:\sampledata\vehicle1_09142014_Copy.csv"

To delete a file, use the following command:

Remove-AzureRmDataLakeStoreItem -AccountName $dataLakeStoreName -Paths $myrootdir\mynewdirectory\vehicle1_09142014_Copy.csv

When prompted, enter Y to delete the item. If you have more than one file to delete, you can provide all the paths separated by comma.

Remove-AzureRmDataLakeStoreItem -AccountName $dataLakeStoreName -Paths $myrootdir\mynewdirectory\vehicle1_09142014.csv, $myrootdir\mynewdirectoryvehicle1_09142014_Copy.csv

Delete your Azure Data Lake Store account

Use the following command to delete your Data Lake Store account.

Remove-AzureRmDataLakeStoreAccount -Name $dataLakeStoreName

When prompted, enter Y to delete the account.

Next steps