question

sachingupta-1921 avatar image
0 Votes"
sachingupta-1921 asked AnnuKumari-MSFT commented

How to delete all Azure data factory pipeline and associated Dataflows, Datasets, Linked Services and Pipeline triggers from UAT environment

Hello,

I am performing Data Factory pipeline deployments from Development environment to UAT using ARM templates. I need to understand below situations while doing the deployment

1) After first time deployment, if I delete/update any data factory pipeline from/in development environment and then do the deployment into UAT, does the same pipeline gets deleted/updated automatically from/in UAT as well?

2) Is there any DevOps pipeline task which can delete all the ADF pipelines and associated datasets, dataflows, linked services and triggers from UAT before doing the next round of deployments from DEV to UAT?

Kindly provide me some clarification on these questions as I am doing this deployments first time.

Appreciate all the help!

azure-data-factoryazure-webapps-content-deployment
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

AnnuKumari-MSFT avatar image
0 Votes"
AnnuKumari-MSFT answered AnnuKumari-MSFT commented

Hi @sachingupta-1921 ,

As I understand your question, it seems you want to know how does the deployment works. Does it delete the existing pipeline from UAT and copy the new one from dev or does it update the pipeline in UAT. Also, you want to know if there is any workaround for deleting all the objects in ADF in higher environment. Please let me know if my understanding about your query is not correct.

Azure Data Factory utilizes Azure Resource Manager templates to store the configuration of your various ADF entities (pipelines, datasets, data flows, and so on).

  1. Once you publish your changes in ADF dev master branch , adf_publish branch gets updated with new template .json file. The same is being pulled by Azure Resource Manager during the deployment . So , basically the UAT ARM template gets overwritten by the new template generated by adf_publish branch.

  2. You can write powershell script and call Run Inline powershell task in your release pipeline to perform this action :
    a. Get the information of all the pipelines using this command : Get-AzureRmDataFactoryPipeline . Click here to check document.
    b. Loop through each of the pipelines and use the command : Remove-AzureRmDataFactoryPipeline to delete all the pipelines from your data
    factory. Click here to check document.


Similarly , for deleting linked service ,check this: Remove-AzureRmDataFactoryLinkedService
For deleting datasets, check this : Remove-AzureRmDataFactoryDataset

Hope this will help. Please let us know if any further queries.


  • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you.
    Original posters help the community find answers faster by identifying the correct answer. Here is how

  • Want a reminder to come back and check responses? Here is how to subscribe to a notification

  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators


· 4
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @sachingupta-1921 ,
Just checking in to see if the above answer helped. Please do consider clicking Accept Answer and Up-Vote for the same as accepted answers help community as well. If you have any further query do let us know.

0 Votes 0 ·

Hi @sachingupta-1921 ,
Just following up to see if the above answer helped. Please do consider clicking Accept Answer and Up-Vote for the same as accepted answers help community as well. If you have any further query do let us know.

0 Votes 0 ·

@AnnuKumari-MSFT ... yes your answer helped to resolve my problem. I do have one more question which I posted here https://docs.microsoft.com/en-us/answers/questions/832984/duplicate-override-arm-template-parameters-while-d.html. Would you be able to take a look as this is also related to the pipeline deployment?

1 Vote 1 ·
Show more comments