Get started with Azure Data Lake Store using .NET SDK

Learn how to use the Azure Data Lake Store .NET SDK to perform basic operations such as create folders, upload and download data files, etc. For more information about Data Lake, see Azure Data Lake Store.

Prerequisites

  • Visual Studio 2013, 2015, or 2017. The instructions below use Visual Studio 2015 Update 2.

  • An Azure subscription. See Get Azure free trial.

  • Azure Data Lake Store account. For instructions on how to create an account, see Get started with Azure Data Lake Store

  • Create an Azure Active Directory Application. You use the Azure AD application to authenticate the Data Lake Store application with Azure AD. There are different approaches to authenticate with Azure AD, which are end-user authentication or service-to-service authentication. For instructions and more information on how to authenticate, see End-user authentication or Service-to-service authentication.

Create a .NET application

  1. Open Visual Studio and create a console application.
  2. From the File menu, click New, and then click Project.
  3. From New Project, type or select the following values:

    Property Value
    Category Templates/Visual C#/Windows
    Template Console Application
    Name CreateADLApplication
  4. Click OK to create the project.
  5. Add the Nuget packages to your project.

    1. Right-click the project name in the Solution Explorer and click Manage NuGet Packages.
    2. In the Nuget Package Manager tab, make sure that Package source is set to nuget.org and that Include prerelease check box is selected.
    3. Search for and install the following NuGet packages:

      • Microsoft.Azure.Management.DataLake.Store - This tutorial uses v2.1.3-preview.
      • Microsoft.Rest.ClientRuntime.Azure.Authentication - This tutorial uses v2.2.12.

        Add a Nuget source

    4. Close the Nuget Package Manager.
  6. Open Program.cs, delete the existing code, and then include the following statements to add references to namespaces.

     using System;
     using System.IO;
     using System.Security.Cryptography.X509Certificates; // Required only if you are using an Azure AD application created with certificates
     using System.Threading;
    
     using Microsoft.Azure.Management.DataLake.Store;
     using Microsoft.Azure.Management.DataLake.Store.Models;
     using Microsoft.IdentityModel.Clients.ActiveDirectory;
     using Microsoft.Rest.Azure.Authentication;
    
  7. Declare the variables as shown below, and provide the values for Data Lake Store name and the resource group name that already exist. Also, make sure the local path and file name you provide here must exist on the computer. Add the following code snippet after the namespace declarations.

     namespace SdkSample
     {
         class Program
         {
             private static DataLakeStoreAccountManagementClient _adlsClient;
             private static DataLakeStoreFileSystemManagementClient _adlsFileSystemClient;
    
             private static string _adlsAccountName;
             private static string _resourceGroupName;
             private static string _location;
             private static string _subId;
    
             private static void Main(string[] args)
             {
                 _adlsAccountName = "<DATA-LAKE-STORE-NAME>"; // TODO: Replace this value with the name of your existing Data Lake Store account.
                 _resourceGroupName = "<RESOURCE-GROUP-NAME>"; // TODO: Replace this value with the name of the resource group containing your Data Lake Store account.
                 _location = "East US 2";
                 _subId = "<SUBSCRIPTION-ID>";
    
                 string localFolderPath = @"C:\local_path\"; // TODO: Make sure this exists and can be overwritten.
                 string localFilePath = Path.Combine(localFolderPath, "file.txt"); // TODO: Make sure this exists and can be overwritten.
                 string remoteFolderPath = "/data_lake_path/";
                 string remoteFilePath = Path.Combine(remoteFolderPath, "file.txt");
             }
         }
     }
    

In the remaining sections of the article, you can see how to use the available .NET methods to perform operations such as authentication, file upload, etc.

Authentication

Use this with an existing Azure AD native application to authenticate your application interactively, which means you will be prompted to enter your Azure credentials.

For ease of use, the snippet below uses default values for client ID and redirect URI that will work with any Azure subscription. To help you complete this tutorial faster, we recommend you use this approach. In the snippet below, just provide the value for your tenant ID. You can retrieve it using the instructions provided at Create an Active Directory Application.

// User login via interactive popup
// Use the client ID of an existing AAD Web application.
SynchronizationContext.SetSynchronizationContext(new SynchronizationContext());
var tenant_id = "<AAD_tenant_id>"; // Replace this string with the user's Azure Active Directory tenant ID
var nativeClientApp_clientId = "1950a258-227b-4e31-a9cf-717495945fc2";
var activeDirectoryClientSettings = ActiveDirectoryClientSettings.UsePromptOnly(nativeClientApp_clientId, new Uri("urn:ietf:wg:oauth:2.0:oob"));
var creds = UserTokenProvider.LoginWithPromptAsync(tenant_id, activeDirectoryClientSettings).Result;

A couple of things to know about this snippet above:

  • To help you complete the tutorial faster, this snippet uses an an Azure AD domain and client ID that is available by default for all Azure subscriptions. So, you can use this snippet as-is in your application.
  • However, if you do want to use your own Azure AD domain and application client ID, you must create an Azure AD native application and then use the Azure AD tenant ID, client ID, and redirect URI for the application you created. See Create an Active Directory Application for end-user authentication with Data Lake Store for instructions.

If you are using service-to-service authentication with client secret

The following snippet can be used to authenticate your application non-interactively, using the client secret / key for an application / service principal. Use this with an existing Azure AD "Web App" Application. For instructions on how to create the Azure AD web application and how to retrieve the client ID and client secret required in the snippet below, see Create an Active Directory Application for service-to-service authentication with Data Lake Store.

// Service principal / appplication authentication with client secret / key
// Use the client ID of an existing AAD "Web App" application.
SynchronizationContext.SetSynchronizationContext(new SynchronizationContext());

var domain = "<AAD-directory-domain>";
var webApp_clientId = "<AAD-application-clientid>";
var clientSecret = "<AAD-application-client-secret>";
var clientCredential = new ClientCredential(webApp_clientId, clientSecret);
var creds = await ApplicationTokenProvider.LoginSilentAsync(domain, clientCredential);

If you are using service-to-service authentication with certificate

As a third option, the following snippet can be used to authenticate your application non-interactively, using the certificate for an Azure Active Directory application / service principal. Use this with an existing Azure AD Application with certificates.

// Service principal / application authentication with certificate
// Use the client ID and certificate of an existing AAD "Web App" application.
SynchronizationContext.SetSynchronizationContext(new SynchronizationContext());

var domain = "<AAD-directory-domain>";
var webApp_clientId = "<AAD-application-clientid>";
var clientCert = <AAD-application-client-certificate>
var clientAssertionCertificate = new ClientAssertionCertificate(webApp_clientId, clientCert);
var creds = await ApplicationTokenProvider.LoginSilentWithCertificateAsync(domain, clientAssertionCertificate);

Create client objects

The following snippet creates the Data Lake Store account and filesystem client objects, which are used to issue requests to the service.

// Create client objects and set the subscription ID
_adlsClient = new DataLakeStoreAccountManagementClient(creds) { SubscriptionId = _subId };
_adlsFileSystemClient = new DataLakeStoreFileSystemManagementClient(creds);

List all Data Lake Store accounts within a subscription

The following snippet lists all Data Lake Store accounts within a given Azure subscription.

// List all ADLS accounts within the subscription
public static async Task<List<DataLakeStoreAccount>> ListAdlStoreAccounts()
{
    var response = await _adlsClient.Account.ListAsync();
    var accounts = new List<DataLakeStoreAccount>(response);

    while (response.NextPageLink != null)
    {
        response = _adlsClient.Account.ListNext(response.NextPageLink);
        accounts.AddRange(response);
    }

    return accounts;
}

Create a directory

The following snippet shows a CreateDirectory method that you can use to create a directory within a Data Lake Store account.

// Create a directory
public static async Task CreateDirectory(string path)
{
    await _adlsFileSystemClient.FileSystem.MkdirsAsync(_adlsAccountName, path);
}

Upload a file

The following snippet shows an UploadFile method that you can use to upload files to a Data Lake Store account.

// Upload a file
public static void UploadFile(string srcFilePath, string destFilePath, bool force = true)
{
    _adlsFileSystemClient.FileSystem.UploadFile(_adlsAccountName, srcFilePath, destFilePath, overwrite:force);
}

The SDK supports recursive upload and download between a local file path and a Data Lake Store file path.

Get file or directory info

The following snippet shows a GetItemInfo method that you can use to retrieve information about a file or directory available in Data Lake Store.

// Get file or directory info
public static async Task<FileStatusProperties> GetItemInfo(string path)
{
    return await _adlsFileSystemClient.FileSystem.GetFileStatusAsync(_adlsAccountName, path).FileStatus;
}

List file or directories

The following snippet shows a ListItem method that can use to list the file and directories in a Data Lake Store account.

// List files and directories
public static List<FileStatusProperties> ListItems(string directoryPath)
{
    return _adlsFileSystemClient.FileSystem.ListFileStatus(_adlsAccountName, directoryPath).FileStatuses.FileStatus.ToList();
}

Concatenate files

The following snippet shows a ConcatenateFiles method that you use to concatenate files.

// Concatenate files
public static Task ConcatenateFiles(string[] srcFilePaths, string destFilePath)
{
    await _adlsFileSystemClient.FileSystem.ConcatAsync(_adlsAccountName, destFilePath, srcFilePaths);
}

Append to a file

The following snippet shows a AppendToFile method that you use append data to a file already stored in a Data Lake Store account.

// Append to file
public static async Task AppendToFile(string path, string content)
{
    using (var stream = new MemoryStream(Encoding.UTF8.GetBytes(content)))
    {
        await _adlsFileSystemClient.FileSystem.AppendAsync(_adlsAccountName, path, stream);
    }
}

Download a file

The following snippet shows a DownloadFile method that you use to download a file from a Data Lake Store account.

// Download file
public static void DownloadFile(string srcFilePath, string destFilePath)
{
     _adlsFileSystemClient.FileSystem.DownloadFile(_adlsAccountName, srcFilePath, destFilePath);
}

Next steps