Azure HDInsight libraries for .NET

Overview

The HDInsight Service .NET SDK provides classes that relate to the creation, configuration, submission, and monitoring of Hadoop jobs managed by an Azure HDInsight Service. In addition, it provides classes to manage Azure subscriptions using the HDInsight Service and to configure the clusters, storage accounts, and other assets associated with the HDInsight clusters that are managed by an Azure subscription.

Management libraries

Jobs

Use the Azure HDInsight client SDK to create, manage, and monitor jobs on a Hadoop cluster.

Install the NuGet package directly from the Visual Studio Package Manager console or with the .NET Core CLI.

Visual Studio Package Manager

Install-Package Microsoft.Azure.Management.HDInsight.Job
dotnet add package Microsoft.Azure.Management.HDInsight.Job

Code Example

This example runs a Hive job in a Hadoop cluster.

HDInsightJobManagementClient managementClient = new HDInsightJobManagementClient(clusterUri, credentials);

Dictionary<string, string> defines = new Dictionary<string, string> {
    { "hive.execution.engine", "tez" },
    { "hive.exec.reducers.max", "1" }
};
List<string> arguments = new List<string> { { "argA" }, { "argB" } };
HiveJobSubmissionParameters parameters = new HiveJobSubmissionParameters
{
    Query = "SHOW TABLES",
    Defines = defines,
    Arguments = arguments
};

JobSubmissionResponse jobResponse = managementClient.JobManagement.SubmitHiveJob(parameters);

HDInsight

Use the Azure HDInsight management SDK to create, manage, start, stop, and scale Hadoop clusters.

Install the NuGet package directly from the Visual Studio Package Manager console or with the .NET Core CLI.

Visual Studio Package Manager

Install-Package Microsoft.Azure.Management.HDInsight
dotnet add package Microsoft.Azure.Management.HDInsight

Code Example

This example creates an HDInsight two node Linux Hadoop cluster with an existing Azure Blob Storage.

HDInsightManagementClient managementClient = new HDInsightManagementClient(authToken);
// Set parameters for the new cluster
ClusterCreateParameters parameters = new ClusterCreateParameters
{
    ClusterSizeInNodes = 2,
    UserName = "admin",
    Password = "<Enter HTTP User Password>",
    ClusterType = "Hadoop",
    OSType = OSType.Linux,
    Version = "3.5",
    // Use an Azure storage account as the default storage
    DefaultStorageInfo = new AzureStorageInfo("<StorageAccount>", "<StorageKey>", "<BlobContainerName>"),
    Location = "EAST US 2",
    SshUserName = "sshuser",
    SshPassword = "<Enter SSH User Password>",
};

// Create the cluster
managementClient.Clusters.Create("<ExistingResourceGroupName>", "<NewClusterName>", parameters);

Samples

View the complete list of Azure SQL Database samples.