Del via

az synapse spark job

  • Reference

Manage Synapse Spark batch jobs.

Commands

Name Description Type Status
az synapse spark job cancel

Cancel a Spark job.

 Core GA
az synapse spark job list

List all Spark jobs.

 Core GA
az synapse spark job show

Get a Spark job.

 Core GA
az synapse spark job submit

Submit a Spark job.

 Core GA

az synapse spark job cancel

Edit

Cancel a Spark job.

az synapse spark job cancel --livy-id
                            --spark-pool-name
                            --workspace-name
                            [--yes]

Examples

Cancel a Spark job.

az synapse spark job cancel --livy-id 1 --workspace-name testsynapseworkspace --spark-pool-name testsparkpool

Required Parameters

--livy-id

The id of the Spark job.

--spark-pool-name

The name of the Spark pool.

--workspace-name

The name of the workspace.

Optional Parameters

--yes -y

Do not prompt for confirmation.

Default value: False
Global Parameters
--debug

Increase logging verbosity to show all debug logs.

--help -h

Show this help message and exit.

--only-show-errors

Only show errors, suppressing warnings.

--output -o

Output format.

Accepted values: json, jsonc, none, table, tsv, yaml, yamlc
Default value: json
--query

JMESPath query string. See http://jmespath.org/ for more information and examples.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--verbose

Increase logging verbosity. Use --debug for full debug logs.

az synapse spark job list

Edit

List all Spark jobs.

az synapse spark job list --spark-pool-name
                          --workspace-name
                          [--from-index]
                          [--size]

Examples

List all Spark jobs.

az synapse spark job list --workspace-name testsynapseworkspace --spark-pool-name testsparkpool

Required Parameters

--spark-pool-name

The name of the Spark pool.

--workspace-name

The name of the workspace.

Optional Parameters

--from-index

Optional parameter specifying which index the list should begin from.

--size

The size of the returned list.By default it is 20 and that is the maximum.

Global Parameters
--debug

Increase logging verbosity to show all debug logs.

--help -h

Show this help message and exit.

--only-show-errors

Only show errors, suppressing warnings.

--output -o

Output format.

Accepted values: json, jsonc, none, table, tsv, yaml, yamlc
Default value: json
--query

JMESPath query string. See http://jmespath.org/ for more information and examples.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--verbose

Increase logging verbosity. Use --debug for full debug logs.

az synapse spark job show

Edit

Get a Spark job.

az synapse spark job show --livy-id
                          --spark-pool-name
                          --workspace-name

Examples

Get a Spark job.

az synapse spark job show --livy-id 1 --workspace-name testsynapseworkspace --spark-pool-name testsparkpool

Required Parameters

--livy-id

The id of the Spark job.

--spark-pool-name

The name of the Spark pool.

--workspace-name

The name of the workspace.

Global Parameters
--debug

Increase logging verbosity to show all debug logs.

--help -h

Show this help message and exit.

--only-show-errors

Only show errors, suppressing warnings.

--output -o

Output format.

Accepted values: json, jsonc, none, table, tsv, yaml, yamlc
Default value: json
--query

JMESPath query string. See http://jmespath.org/ for more information and examples.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--verbose

Increase logging verbosity. Use --debug for full debug logs.

az synapse spark job submit

Edit

Submit a Spark job.

az synapse spark job submit --executor-size {Large, Medium, Small}
                            --executors
                            --main-definition-file
                            --name
                            --spark-pool-name
                            --workspace-name
                            [--archives]
                            [--arguments]
                            [--configuration]
                            [--language {CSharp, PySpark, Python, Scala, Spark, SparkDotNet}]
                            [--main-class-name]
                            [--reference-files]
                            [--tags]

Examples

Submit a Java Spark job.

az synapse spark job submit --name WordCount_Java --workspace-name testsynapseworkspace \
--spark-pool-name testsparkpool \
--main-definition-file abfss://testfilesystem@testadlsgen2.dfs.core.windows.net/samples/java/wordcount/wordcount.jar \
--main-class-name WordCount \
--arguments abfss://testfilesystem@testadlsgen2.dfs.core.windows.net/samples/java/wordcount/shakespeare.txt \
abfss://testfilesystem@testadlsgen2.dfs.core.windows.net/samples/java/wordcount/result/ \
--executors 2 --executor-size Small

Required Parameters

--executor-size

The executor size.

Accepted values: Large, Medium, Small
--executors

The number of executors.

--main-definition-file

The main file used for the job.

--name -n

The Spark job name.

--spark-pool-name

The name of the Spark pool.

--workspace-name

The name of the workspace.

Optional Parameters

--archives

The array of archives.

--arguments

Optional arguments to the job (Note: please use storage URIs for file arguments).

--configuration

The configuration of Spark job.

--language

The Spark job language.

Accepted values: CSharp, PySpark, Python, Scala, Spark, SparkDotNet
Default value: Scala
--main-class-name

The fully-qualified identifier or the main class that is in the main definition file.

--reference-files

Additional files used for reference in the main definition file.

--tags

Space-separated tags: key[=value] [key[=value] ...]. Use "" to clear existing tags.

Global Parameters
--debug

Increase logging verbosity to show all debug logs.

--help -h

Show this help message and exit.

--only-show-errors

Only show errors, suppressing warnings.

--output -o

Output format.

Accepted values: json, jsonc, none, table, tsv, yaml, yamlc
Default value: json
--query

JMESPath query string. See http://jmespath.org/ for more information and examples.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--verbose

Increase logging verbosity. Use --debug for full debug logs.