Get started with Azure Data Lake Store using Azure CLI 2.0

Learn how to use Azure CLI 2.0 to create an Azure Data Lake Store account and perform basic operations such as create folders, upload and download data files, delete your account, etc. For more information about Data Lake Store, see Overview of Data Lake Store.

The Azure CLI 2.0 is Azure's new command-line experience for managing Azure resources. It can be used on macOS, Linux, and Windows. For more information, see Overview of Azure CLI 2.0. You can also look at the Azure Data Lake Store CLI 2.0 reference for a complete list of commands and syntax.

Prerequisites

Before you begin this article, you must have the following:

Authentication

This article uses a simpler authentication approach with Data Lake Store where you log in as an end-user user. The access level to Data Lake Store account and file system is then governed by the access level of the logged in user. However, there are other approaches as well to authenticate with Data Lake Store, which are end-user authentication or service-to-service authentication. For instructions and more information on how to authenticate, see End-user authentication or Service-to-service authentication.

Log in to your Azure subscription

  1. Log into your Azure subscription.

    az login
    

    You get a code to use in the next step. Use a web browser to open the page https://aka.ms/devicelogin and enter the code to authenticate. You are prompted to log in using your credentials.

  2. Once you log in, the window lists all the Azure subscriptions that are associated with your account. Use the following command to use a specific subscription.

    az account set --subscription <subscription id> 
    

Create an Azure Data Lake Store account

  1. Create a new resource group. In the following command, provide the parameter values you want to use. If the location name contains spaces, put it in quotes. For example "East US 2".

    az group create --location "East US 2" --name myresourcegroup
    
  2. Create the Data Lake Store account.

    az dls account create --account mydatalakestore --resource-group myresourcegroup
    

Create folders in a Data Lake Store account

You can create folders under your Azure Data Lake Store account to manage and store data. Use the following command to create a folder called mynewfolder at the root of the Data Lake Store.

az dls fs create --account mydatalakestore --path /mynewfolder --folder

Note

The --folder parameter ensures that the command creates a folder. If this parameter is not present, the command creates an empty file called mynewfolder at the root of the Data Lake Store account.

Upload data to a Data Lake Store account

You can upload data to Data Lake Store directly at the root level or to a folder that you created within the account. The snippets below demonstrate how to upload some sample data to the folder (mynewfolder) you created in the previous section.

If you are looking for some sample data to upload, you can get the Ambulance Data folder from the Azure Data Lake Git Repository. Download the file and store it in a local directory on your computer, such as C:\sampledata.

az dls fs upload --account mydatalakestore --source-path "C:\SampleData\AmbulanceData\vehicle1_09142014.csv" --destination-path "/mynewfolder/vehicle1_09142014.csv"

Note

For the destination, you must specify the complete path including the file name.

List files in a Data Lake Store account

Use the following command to list the files in a Data Lake Store account.

az dls fs list --account mydatalakestore --path /mynewfolder

The output of this should be similar to the following:

[
    {
        "accessTime": 1491323529542,
        "aclBit": false,
        "blockSize": 268435456,
        "group": "1808bd5f-62af-45f4-89d8-03c5e81bac20",
        "length": 1589881,
        "modificationTime": 1491323531638,
        "msExpirationTime": 0,
        "name": "mynewfolder/vehicle1_09142014.csv",
        "owner": "1808bd5f-62af-45f4-89d8-03c5e81bac20",
        "pathSuffix": "vehicle1_09142014.csv",
        "permission": "770",
        "replication": 1,
        "type": "FILE"
    }
]

Rename, download, and delete data from a Data Lake Store account

  • To rename a file, use the following command:

    az dls fs move --account mydatalakestore --source-path /mynewfolder/vehicle1_09142014.csv --destination-path /mynewfolder/vehicle1_09142014_copy.csv
    
  • To download a file, use the following command. Make sure the destination path you specify already exists.

    az dls fs download --account mydatalakestore --source-path /mynewfolder/vehicle1_09142014_copy.csv --destination-path "C:\mysampledata\vehicle1_09142014_copy.csv"
    

    Note

    The command creates the destination folder if it does not exist.

  • To delete a file, use the following command:

    az dls fs delete --account mydatalakestore --path /mynewfolder/vehicle1_09142014_copy.csv
    

    If you want to delete the folder mynewfolder and the file vehicle1_09142014_copy.csv together in one command, use the --recurse parameter

    az dls fs delete --account mydatalakestore --path /mynewfolder --recurse
    

Work with permissions and ACLs for a Data Lake Store account

In this section you learn about how to manage ACLs and permissions using Azure CLI 2.0. For a detailed discussion on how ACLs are implemented in Azure Data Lake Store, see Access control in Azure Data Lake Store.

  • To update the owner of a file/folder, use the following command:

    az dls fs access set-owner --account mydatalakestore --path /mynewfolder/vehicle1_09142014.csv --group 80a3ed5f-959e-4696-ba3c-d3c8b2db6766 --owner 6361e05d-c381-4275-a932-5535806bb323
    
  • To update the permissions for a file/folder, use the following command:

    az dls fs access set-permission --account mydatalakestore --path /mynewfolder/vehicle1_09142014.csv --permission 777
    
  • To get the ACLs for a given path, use the following command:

    az dls fs access show --account mydatalakestore --path /mynewfolder/vehicle1_09142014.csv
    

    The output should be similar to the following:

      {
          "entries": [
          "user::rwx",
          "group::rwx",
          "other::---"
        ],
        "group": "1808bd5f-62af-45f4-89d8-03c5e81bac20",
        "owner": "1808bd5f-62af-45f4-89d8-03c5e81bac20",
        "permission": "770",
        "stickyBit": false
      }
    
  • To set an entry for an ACL, use the following command:

    az dls fs access set-entry --account mydatalakestore --path /mynewfolder --acl-spec user:6360e05d-c381-4275-a932-5535806bb323:-w-
    
  • To remove an entry for an ACL, use the following command:

    az dls fs access remove-entry --account mydatalakestore --path /mynewfolder --acl-spec user:6360e05d-c381-4275-a932-5535806bb323
    
  • To remove an entire default ACL, use the following command:

    az dls fs access remove-all --account mydatalakestore --path /mynewfolder --default-acl
    
  • To remove an entire non-default ACL, use the following command:

    az dls fs access remove-all --account mydatalakestore --path /mynewfolder
    

Delete a Data Lake Store account

Use the following command to delete a Data Lake Store account.

az dls account delete --account mydatalakestore

When prompted, enter Y to delete the account.

Next steps