How to delete all Azure data factory pipeline and associated Dataflows, Datasets, Linked Services and Pipeline triggers from UAT environment

Question

Hello,

I am performing Data Factory pipeline deployments from Development environment to UAT using ARM templates. I need to understand below situations while doing the deployment

1) After first time deployment, if I delete/update any data factory pipeline from/in development environment and then do the deployment into UAT, does the same pipeline gets deleted/updated automatically from/in UAT as well?

2) Is there any DevOps pipeline task which can delete all the ADF pipelines and associated datasets, dataflows, linked services and triggers from UAT before doing the next round of deployments from DEV to UAT?

Kindly provide me some clarification on these questions as I am doing this deployments first time.

Appreciate all the help!

Answer 1

Hi @sachin gupta ,

As I understand your question, it seems you want to know how does the deployment works. Does it delete the existing pipeline from UAT and copy the new one from dev or does it update the pipeline in UAT. Also, you want to know if there is any workaround for deleting all the objects in ADF in higher environment. Please let me know if my understanding about your query is not correct.

Azure Data Factory utilizes Azure Resource Manager templates to store the configuration of your various ADF entities (pipelines, datasets, data flows, and so on).

1. Once you publish your changes in ADF dev master branch , adf_publish branch gets updated with new template .json file. The same is being pulled by Azure Resource Manager during the deployment . So , basically the UAT ARM template gets overwritten by the new template generated by adf_publish branch.
2. You can write powershell script and call Run Inline powershell task in your release pipeline to perform this action :
   a. Get the information of all the pipelines using this command : Get-AzureRmDataFactoryPipeline . Click here to check document.
   b. Loop through each of the pipelines and use the command : Remove-AzureRmDataFactoryPipeline to delete all the pipelines from your data
   factory. Click here to check document.

Similarly , for deleting linked service ,check this: Remove-AzureRmDataFactoryLinkedService
For deleting datasets, check this : Remove-AzureRmDataFactoryDataset

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you.
  Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators