New-AzureRmHDInsightStreamingMapReduceJobDefinition

Reference

Module:: AzureRM.HDInsight

Creates a Streaming MapReduce job object.

Warning

The AzureRM PowerShell module has been officially deprecated as of February 29, 2024. Users are advised to migrate from AzureRM to the Az PowerShell module to ensure continued support and updates.

Although the AzureRM module may still function, it's no longer maintained or supported, placing any continued use at the user's discretion and risk. Please refer to our migration resources for guidance on transitioning to the Az module.

Syntax

New-AzureRmHDInsightStreamingMapReduceJobDefinition
   [-Arguments <String[]>]
   [-File <String>]
   [-Files <String[]>]
   [-StatusFolder <String>]
   [-CommandEnvironment <Hashtable>]
   [-Defines <Hashtable>]
   -InputPath <String>
   [-Mapper <String>]
   [-OutputPath <String>]
   [-Reducer <String>]
   [-DefaultProfile <IAzureContextContainer>]
   [<CommonParameters>]

Description

The New-AzureRmHDInsightStreamingMapReduceJobDefinition cmdlet defines a Streaming MapReduce job object for use with an Azure HDInsight cluster.

Examples

Example 1: Create a Streaming MapReduce job definition

PS C:\># Cluster info
PS C:\>$clusterName = "your-hadoop-001"
PS C:\>$clusterCreds = Get-Credential

# Streaming MapReduce job details
PS C:\>$statusFolder = "tempStatusFolder/"
PS C:\>$query = "SHOW TABLES"

PS C:\>New-AzureRmHDInsightStreamingMapReduceJobDefinition -StatusFolder $statusFolder `
            -Query $query `
        | Start-AzureRmHDInsightJob `
            -ClusterName $clusterName `
            -ClusterCredential $clusterCreds

This command creates a Streaming MapReduce job definition.

Parameters

-Arguments

Specifies an array of arguments for the job. The arguments are passed as command-line arguments to each task.

Type:	String[]
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-CommandEnvironment

Specifies an array of command-line environment variables to set when a job runs on worker nodes.

Type:	Hashtable
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-DefaultProfile

The credentials, account, tenant, and subscription used for communication with azure

Type:	IAzureContextContainer
Aliases:	AzureRmContext, AzureCredential
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-Defines

Specifies Hadoop configuration values to set for when the job runs.

Type:	Hashtable
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-File

Specifies the path to a file that contains a query to run. You can use this parameter instead of the Query parameter.

Type:	String
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-Files

Specifies a collection of files that are associated with a Hive job.

Type:	String[]
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-InputPath

Specifies the path to the input files.

Type:	String
Position:	Named
Default value:	None
Required:	True
Accept pipeline input:	False
Accept wildcard characters:	False

-Mapper

Specifies a Mapper file name.

Type:	String
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-OutputPath

Specifies the path for the job output.

Type:	String
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-Reducer

Specifies a Reducer file name.

Type:	String
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-StatusFolder

Specifies the location of the folder that contains standard outputs and error outputs for a job.

Type:	String
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

Inputs

None

Outputs

AzureHDInsightStreamingMapReduceJobDefinition

Start-AzureRmHDInsightJob