question

GeetanjaliPuri-6444 avatar image
0 Votes"
GeetanjaliPuri-6444 asked GeetanjaliPuri-6444 commented

Edit Apache Spark application job definition through Azure Devops

We have created a Apache Spark Job Definition in Azure Synapse & want to know if we can edit its configuration using Azure Devops pipeline.

azure-synapse-analytics
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

SamaraSoucy-MSFT avatar image
0 Votes"
SamaraSoucy-MSFT answered GeetanjaliPuri-6444 commented

Yes, though how depends on the scope of your ADO deployment.

If you are just looking to update a few items but leave the rest outside of the pipeline, then you can call the Azure REST API to update the Spark job.

PUT exampleWorkspace.dev.azuresynapse.net/sparkJobDefinitions/exampleSparkJobDefinition?api-version=2019-06-01-preview

Body:
{
"properties": {
"description": "A sample spark job definition",
"targetBigDataPool": {
"referenceName": "exampleBigDataPool",
"type": "BigDataPoolReference"
},
"requiredSparkVersion": "2.4",
"jobProperties": {
"name": "exampleSparkJobDefinition",
"file": "abfss://test@test.dfs.core.windows.net/artefacts/sample.jar",
"className": "dev.test.tools.sample.Main",
"conf": {},
"args": [
"exampleArg"
],
"jars": [],
"pyFiles": [],
"files": [],
"archives": [],
"driverMemory": "28g",
"driverCores": 4,
"executorMemory": "28g",
"executorCores": 4,
"numExecutors": 2
}
}
}

Alternatively, if you want to fully integrate your Synapse workspace with ADO, then it has CI/CD capability built in. You can look at the current instructions and requirements for this to work in the doc: https://docs.microsoft.com/en-us/azure/synapse-analytics/cicd/continuous-integration-deployment


· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

TThanks for the update! Is it possible to have a short call to understand this better?

0 Votes 0 ·

We can arrange that. Please send an email to AzCommunity@microsoft.com to my attention with a link to this thread. If you could also include what time zone you are in and if you have preferences for specific times we can work on setting up a call with myself or one of my teammates.

0 Votes 0 ·

Sure, thanks!

0 Votes 0 ·