Hello Guys, I am trying to implement CICD pipeline for ADF. I have only two adf in same resource group and both need a self hosted integration services to connect on premises database. I was able to do all steps of cicd pipeline but during arm template deployment the process is getting terminated because self hosted integration runtime name is different and it needs same name to encrypt the connection. We cannot give same name to IR because they both share same resource. IR name is not parameterized so it appears arm template not in arm template parameters file. So I can't override the parameter. There is solution available in msft documents to follow ternary adf architecture. So this way third adf will only host IR and the rest two can share the resources and have same name. Is there any way I can avoid this. Any help or clue is appreciated. Thanks!

Hello @Sabyasachi Dash , welcome to Microsoft Q&A and thank you for your question. Also, thank you for adding tag azure-data-factory . The tag adfs is for something unrelated. I am not confident there is an acceptable solution beyond the ternary architectures, but I am willing to give it a try. Let me tell you why it is difficult. Even if you were able to give both IR the same name, you would still run into trouble. This is because a Linked IR and Shared IR are different. One is the 'owner' of the SHIR, and the other is the 'borrower'. If you deployed one over the other, you would end up with 2 'owners' or 2 'borrowers'. There can only be 1 owner, and a borrower needs an owner to point to. My colleague has a workaround where he edits part of the template. I can look into whether the IR name can be parameterized, if you like, but as explained above, you would still run into another issue. p.s. I meant this as a comment, but Q&A is giving me a hard time.

Can you please detail me about the steps for setting up ternary architecture. I am looking for setting up shared integration runtime and came across this post. Would like to understand if there are any limitations with Shared self hosted Integration Runtime and is this supported through Azure DevOps CI/CD pipeline.

CICD deployment for ADF with shared self hosted IR

Accepted answer

MartinJaffer-MSFT 26,026 Reputation points

2020-09-15T00:38:02.46+00:00

Hello @Sabyasachi Dash , welcome to Microsoft Q&A and thank you for your question. Also, thank you for adding tag azure-data-factory. The tag adfs is for something unrelated.

I am not confident there is an acceptable solution beyond the ternary architectures, but I am willing to give it a try. Let me tell you why it is difficult.

Even if you were able to give both IR the same name, you would still run into trouble. This is because a Linked IR and Shared IR are different. One is the 'owner' of the SHIR, and the other is the 'borrower'. If you deployed one over the other, you would end up with 2 'owners' or 2 'borrowers'. There can only be 1 owner, and a borrower needs an owner to point to.

My colleague has a workaround where he edits part of the template.
I can look into whether the IR name can be parameterized, if you like, but as explained above, you would still run into another issue.

p.s. I meant this as a comment, but Q&A is giving me a hard time.
Please sign in to rate this answer.

1 person found this answer helpful.
MartinJaffer-MSFT 26,026 Reputation points

2020-09-16T19:25:03.59+00:00

@Sabyasachi Dash did my response answer your question?

Any work-around would involve some modification to your process, whether that be manual editing of ARM templates, a change to your dev-ops process, or adding a git-ignore. Without knowing your process, I can't help.

If my previous post answered your question, please mark as accepted answer.
Otherwise let me know if you have further questions.

Sabyasachi Dash 21 Reputation points

2020-09-17T22:16:13.25+00:00

@MartinJaffer-MSFT Yeah i tried ternary architecture and i was able to solve the aforementioned issue. I used a third ADF to host SHIR. Two other ADF's are linked with ADF which hosts the SHIR. It allowed me to keep same name.
So i was able to deploy from dev to prd.

In the documentation there is section to fully automate the template override.Yet to try.

Another doubt, i am deploying from DEV to PRD ,both have Azure git enabled so when i am deploying, it doesn't commit to master branch of ADF. Instead it commit changes to the Data factory section(PIC below) which is not git enabled. Is this expected behavior ? If someone want to implement git in all environment along with CICD, the above mentioned scenario wont allow.

Thank you for your help!

Thanks.

Sabyasachi Dash 21 Reputation points

2020-09-17T22:18:18.067+00:00

I read some blog about manual template override and deployment. It didn't fit to my solution as i needed my whole pipeline deployment to be automated.

MartinJaffer-MSFT 26,026 Reputation points

2020-09-18T19:06:45.703+00:00

Hmm interesting, @Sabyasachi Dash .

The Data factory section in your picture tries to reflect the "live" version, the one already published, the one triggers create runs on.
Publishing is the act of taking the (collaboration)master branch and pushing it to the Data factory section and making it live.

Typically, only the development environment factory is git enabled. The master branch is published in the development factory. When it does, an artifact is created in the adf_publish branch. If all is acceptable, then this version gets deployed to next environment factory.

I am not sure how things should work if all environments are git enabled. Can you describe to me how you would use it?
Would each repo be identical? How does enabling git in each environment help?
Please help me understand the use case.

Thank you.

Sabyasachi Dash 21 Reputation points

2020-09-18T20:28:47.92+00:00

@MartinJaffer-MSFT Honestly we made mistake initially enabling GIT for all environment. We though of using it as versioning tool and though CDCI would complement it. I totally agree with you that we should enable git on dev deploy changes from DEV to PRD through CDCI. There was no such advantage or architecture requirement as such to enable GIT in DEV and PRD system.

Initially we were doing a POC but as it became full pledged solution so we opted for DEV and PRD arch. While doing this we thought of why not we use CDCI and it will streamline our process. We had already GIT enabled for PRD and enabled GIT for the new DEV system commissioned. The GIT was enabled in both thinking we can use CDCI with GIT enabled in both environment. But later came to realize it is best not to have GIT enabled in both Env and Devops solution wont allow this for ADF. This is just a bad arch planning on our end.

I asked you this question out of curiosity if we can do this or not. Hope this clears.

Thanks.
Sign in to comment

CICD deployment for ADF with shared self hosted IR

1 additional answer