question

TimHellegers-2738 avatar image
0 Votes"
TimHellegers-2738 asked SaurabhSharma-msft commented

How to structure ADF when having multiple data projects?

Hi all,

What is this best way to organize ADF when there are multiple projects that are not related?

We now have one project with two ADF components (ADF-DEV and ADF-PROD). We will be using GitHub to control the versions and updates to production. But in the future we want to use ADF for multiple projects that are not related.
Is it better to create two additional ADF components (DEV and PROD) for each project or structure them in the excisting components?

I cannot find a proper way to this question on the internet.

Best,
Tim

azure-data-factory
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

SaurabhSharma-msft avatar image
0 Votes"
SaurabhSharma-msft answered SaurabhSharma-msft commented

Hi @timhellegers-2738,

Thanks for using Microsoft Q&A !!

In my opinion, you could create separate ADF's for different projects as those are separate. Also, I suggest you to establish a CI/CD Pipeline for each your Project ADFs which will help you move pipelines easily from one environment(dev/prod) to another instead of using two different ADF's (ADF-DEV, ADF-PROD), one for development and one ADF for production.
Please refer to the documentation for more details. This way you will not affect pipeline of any other projects.

Please let me if this helps.

Thanks
Saurabh


Please do not forget to "Accept the answer" wherever the information provided helps you to help others in the community.

· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thank you for your answer @SaurabhSharma-msft . Using different ADF's for projects makes sense. So if I'm correct, the CI/CD pipeline manages the development seperatly. What if we need to test the pipelines before merging them with the main branch? Is this managed by using CI/CD as well, or do we need a seperate ADF there?
We are planning on creating a seperate database so we do not work with real-time data for our testing/development.

Best,
Tim

0 Votes 0 ·

@timhellegers-2738 @Samy-7940, No, you do not need to manage separate ADF's but when you are satisfied with your changes in development environment, you can use the configured CI/CD pipeline to move the required changes to next environment (test/uat) and later to production. So your pipeline life cycle would like below as mentioned in the shared document -
120500-image.png
Please let me know if you have any other questions.

Thanks
Saurabh

1 Vote 1 ·
image.png (165.4 KiB)

Hi Saurabh, I am sorry, not able to get what exactly you are trying to convey here. Could you please elaborate a bit with an example. Thanks

0 Votes 0 ·