question

MuraliKrishna-2396 avatar image
0 Votes"
MuraliKrishna-2396 asked HimanshuSinha-MSFT commented

COPY Activity DIU's not using as per settings

Hello Team,

I have a copy activity which extracts the data from Azure PostgreSQL to Data lake Gen1(CSV), this pipeline extracts more than one billion records daily to data lake, to improve the performance we have increased the DIU's to 30 and Copy parallel property to 8, however the pipeline is using only 4 DIU's and Copy parallels 1, I am wondering why ADF is not using all the DIU's though we set them explicitly.

Can anyone show some light here....

azure-data-factoryazure-database-postgresql
· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@HimanshuSinha-MSFT Thanks for checking and answering, I will check the partition in source and will try to implement in ADF, will that increases the DIU's?

0 Votes 0 ·

I dont think so

0 Votes 0 ·

Hello Hello @MuraliKrishna-2396,

  • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how

  • Want a reminder to come back and check responses? Here is how to subscribe to a notification

  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

0 Votes 0 ·

1 Answer

HimanshuSinha-MSFT avatar image
0 Votes"
HimanshuSinha-MSFT answered

Hello @MuraliKrishna-2396,
Thanks for the ask and using Microsoft Q&A platform .
I think you should look at the how the data is partioned on the source side . Have you tried to set the partition and see if that helps .?

133645-image.png



Please do read about the best practices here : https://docs.microsoft.com/en-us/azure/data-factory/connector-azure-sql-database?tabs=data-factory#parallel-copy-from-sql-database


Please do let me know how it goes .
Thanks
Himanshu


  • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how

    • Want a reminder to come back and check responses? Here is how to subscribe to a notification

    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of [Q&A Volunteer Moderators][50]



image.png (17.3 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.