question

Samy-7940 avatar image
0 Votes"
Samy-7940 asked ShaikMaheer-MSFT commented

is there any limitation of writing billions of record from Azure Data Factory

Hi All, I have gone through the same question processing billion records in databricks in to Data warehouse.

https://docs.microsoft.com/en-us/answers/questions/120575/is-there-any-limitation-of-writing-billions-of-rec.html

I tried to find out the approach to do similar task in Azure Data Factory but I couldn't able to find any solution for the same. I will be grateful if you could please explain how you would design a pipeline to process billion records and that should provide a near real time analytics. Thanks a lot.

azure-data-factory
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

ShaikMaheer-MSFT avatar image
1 Vote"
ShaikMaheer-MSFT answered ShaikMaheer-MSFT commented

Hi @Samy-7940 ,

Thank you for posting your query on Microsoft Q&A Platform.

Azure data factory can perform Bigdata ETL operations. You can use Copy activity to perform your data movement. It will support billion rows as well.

Click here to know how to bulk copy multiple tables using Azure data factory
Click here to know how to bulk copy files to database

If your Sink is Azure Synapse then you can use Polybase approach to load data. Click here to know more about this.

Azure Data Factory enriches PolyBase support for loading data into SQL DW. Check below Article for more details.
https://azure.microsoft.com/en-in/updates/azure-data-factory-enriched-polybase-support-for-loading-data-into-sql-dw/

Hope this will help. Please let us know if any further queries. Thank you.


  • Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.

  • Want a reminder to come back and check responses? Here is how to subscribe to a notification.

· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi there,
We have a situation to move transaction records to history tables in azure sql server managed instance database. We have 62M records and it is a direct insert to history from transaction and it takes 18 minutes to process and some times results in timeout. Do u think azure data factory copy activity will do the processes faster. any other suggestions

0 Votes 0 ·
ShaikMaheer-MSFT avatar image ShaikMaheer-MSFT VISWANATHANLAKSHMANAN-9888 ·

Hi @VISWANATHANLAKSHMANAN-9888 ,

Yes Azure Data Factory Copy Activity will help you there. You can configure more DTUs under settings tab of Copy activity to get good performance.

Please check below documentation to know more about Copy activity performance and scalability
https://docs.microsoft.com/en-us/azure/data-factory/copy-activity-performance

Please feel free to open new thread as well to discuss more on this. Thank you.

0 Votes 0 ·