We have created a Apache Spark Job Definition in Azure Synapse & want to know if we can edit its configuration using Azure Devops pipeline.
We have created a Apache Spark Job Definition in Azure Synapse & want to know if we can edit its configuration using Azure Devops pipeline.
Yes, though how depends on the scope of your ADO deployment.
If you are just looking to update a few items but leave the rest outside of the pipeline, then you can call the Azure REST API to update the Spark job.
PUT exampleWorkspace.dev.azuresynapse.net/sparkJobDefinitions/exampleSparkJobDefinition?api-version=2019-06-01-preview
Body:
{
"properties": {
"description": "A sample spark job definition",
"targetBigDataPool": {
"referenceName": "exampleBigDataPool",
"type": "BigDataPoolReference"
},
"requiredSparkVersion": "2.4",
"jobProperties": {
"name": "exampleSparkJobDefinition",
"file": "abfss://test@test.dfs.core.windows.net/artefacts/sample.jar",
"className": "dev.test.tools.sample.Main",
"conf": {},
"args": [
"exampleArg"
],
"jars": [],
"pyFiles": [],
"files": [],
"archives": [],
"driverMemory": "28g",
"driverCores": 4,
"executorMemory": "28g",
"executorCores": 4,
"numExecutors": 2
}
}
}
Alternatively, if you want to fully integrate your Synapse workspace with ADO, then it has CI/CD capability built in. You can look at the current instructions and requirements for this to work in the doc: https://docs.microsoft.com/en-us/azure/synapse-analytics/cicd/continuous-integration-deployment
TThanks for the update! Is it possible to have a short call to understand this better?
We can arrange that. Please send an email to AzCommunity@microsoft.com to my attention with a link to this thread. If you could also include what time zone you are in and if you have preferences for specific times we can work on setting up a call with myself or one of my teammates.
6 people are following this question.