How to Overite parameters for CI/CD pipeline for azure daafactory For Databricks (Workspace Url and Cluster ID)

v-Shubham Kumar Jain 21

Hi, I am doing CI/CD integration for one data factory to another I am Successfully able to Create the release and abe to copy from my Dev to UAT environment I am able to copy my pipelines Triggers and the Link Service
The problem I am facing In just copying Databricks Links Service

As we know we have to override the parameters of our environment, In Databricks Option comes only to override fro Access token
And Databricks require three parameters workspace URL and ClusterID, As there is no option to override these two My workspace URL and Cluster ID is in the production environment is copied of MY Dev environment Token is copied Successfully which result in the Unsuccessful Connection

As you see inside of pic for data bricks three params are required
and Override Template parameter is giving for an only access token

3 answers

HarithaMaddi-MSFT 10,136 Reputation points

2020-10-14T12:31:06.897+00:00

Hi @v-Shubham Kumar Jain ,

Welcome to Microsoft Q&A Platform. Thanks for posting the query.

Please note that those properties can be parameterized from ARM Template by defining parameters for "Domain" and "existingclusterID" properties as below.

Hope this helps! Please let us know for further queries and we will be glad to assist.
Please sign in to rate this answer.
v-Shubham Kumar Jain 21 Reputation points

2020-10-15T09:11:25.893+00:00

Hi HarithaMaddi-MSFT just to be sure if I Understood it properly in my linked service section, I have to go in code the section of the DatabricksLinked Service and Use the Above JSON Code for the parameter in the Linked service By doing that when I will be overriding those parameters I Can see in the Template as an
option to override?

HarithaMaddi-MSFT 10,136 Reputation points

2020-10-15T09:32:18.033+00:00

Hi @v-Shubham Kumar Jain ,

While doing CI/CD, as per my understanding, you might be using ARM templates downloaded from GIT/Data factory as below.

Please parameterize the databricks workspace id and cluster id in the arm_template.json file and the new parameters can be added to "arm_template_parameters.json" file which can be overwritten by deployment release pipelines.

If not using the GIT mode in data factory, please refer to below snap to download ARM Template.

Please suggest if the approach you are using is different and I can suggest accordingly.

v-Shubham Kumar Jain 21 Reputation points

2020-10-15T10:00:41.793+00:00

After publishing data factory code in github In order to override parameters, I am just using in CI/Cd release

we have one ARM Template deployment module l2)inking the Github File 3)Choosing the Link Artifact (Github inked repo ) giving path of ARMTEmplate and ARMTEMPLATEPARMETER.JSON it shows options to override template parameters after selecting ARMTemplatesParameter.json File so this is the way I am following it now not currently changing code of any file

v-Shubham Kumar Jain 21 Reputation points

2020-10-15T10:10:45.29+00:00

did you get what I was trying to explain?

HarithaMaddi-MSFT 10,136 Reputation points

2020-10-15T10:15:33.283+00:00

Hi @v-Shubham Kumar Jain ,

Yes, I understood. I am suggesting to modify the ARMTemplate.json and arm_template_parameters.json from the GIT Branch to add new parameters and map them to domain and existingclusterid as shown to the files. These properties cannot be parameterized from data factory.

Please let me know for further queries and I will be glad to assist.

HarithaMaddi-MSFT 10,136 Reputation points

2020-10-16T07:57:40.197+00:00

Hi @v-Shubham Kumar Jain ,

We have not received a response from you. Please suggest if above details clarified your query and helped in implementing the requirement. Let us know for further queries and we will be glad to assist.

Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members

HarithaMaddi-MSFT 10,136 Reputation points

2020-10-19T08:12:40.49+00:00

Hi @v-Shubham Kumar Jain ,

We still have not heard back from you. Following up to check if above details clarified your query and helped in implementing the requirement. Let us know for further queries and we will be glad to assist.

Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members

v-Shubham Kumar Jain 21 Reputation points

2020-10-20T06:54:28.467+00:00

Hi I tried It i was able to give parameters in ARmtemplateparameter.json bt not able to customize armtemplate.json can you give me specific json for armtemplate.json mdoification thanks

HarithaMaddi-MSFT 10,136 Reputation points

2020-10-20T09:00:08.283+00:00

Thanks @v-Shubham Kumar Jain for reaching out. I made a GIF explaining the same and also attaching the json templates to the thread. If there are still challenges, please send a mail to AzCommunity[at]microsoft[dot]com with subject: "Attn:Haritha - How to Overite parameters for CI/CD pipeline for azure daafactory For Databricks (Workspace Url and Cluster ID)" and we will be glad to assist further.

33544-armtemplateforfactoryjson.txt
33605-armtemplateparametersforfactoryjson.txt

v-Shubham Kumar Jain 21 Reputation points

2020-10-20T11:06:40.193+00:00

Thanks That's really helpful let me try and I will get back to you in some time
Sign in to comment
Chaitanya Kale 1 Reputation point

2021-06-05T07:14:17.647+00:00

@HarithaMaddi-MSFT .. But does this mean we need to manually update ADF publish template and parameter file with DatabricksURL parameter and clusterName parameter each time we publish code from Datafactory UI. Because, these manually added parameter disappear when we publish into adf_publish from datafactory UI
Please sign in to rate this answer.

0 comments No comments
Sign in to comment
Nair, Sreekul 1 Reputation point

2021-08-20T14:58:37.617+00:00

@HarithaMaddi-MSFT - I do have the same concern as that of what @Chaitanya Kale has stated here. Changing the ARM Template Parameter json would be futile once u republish the Dev Master branch to ADF_Publish branch. Is there any other way to go about it?
Please sign in to rate this answer.
Nair, Sreekul 1 Reputation point

2021-08-20T17:03:00.563+00:00

@HarithaMaddi-MSFT - I tested this too and it confirmed my doubt. Whenever we hit Publish in the master branch, any changes made manually to the ARM Template gets overwritten. Any inputs on how to proceed further on the same?

Nair, Sreekul 1 Reputation point

2021-08-24T16:01:54.227+00:00

The way to do this correctly is by going to ADF -> Manage -> ARM Template -> Edit Parameter configuration file.
Please follow the link https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment (section Use custom parameters with the Resource Manager template) for the same.
Sign in to comment