There is some activities running questions: What is the maximum number of source dataset can one activities has? If we set 40 parallel activities in one pipeline, does it run really parallel or one by one actually? If there is one source dataset and 32 sink datasets with different output orc file name with parallel copy equals to 32 (I see 32 is the maximum number I can set), is it really running on 32 threads? is there any way to move a orc block blob to another path without copying and delete old file? Move physical path costs so many time and resource. My requirement is to upload file form local database and use some mechanisms to check the file is complete.

Azure Data Factory Parallel running and Azure Blob Storage move path

Teddy 1

There is some activities running questions:

What is the maximum number of source dataset can one
activities has?
If we set 40 parallel activities in one pipeline, does it run really
parallel or one by one actually?
If there is one source dataset and 32 sink datasets with different output orc file name with parallel copy equals to 32 (I see 32 is the maximum number I can set), is it really running on 32 threads?
is there any way to move a orc block blob to another path without copying and delete old file? Move physical path costs so many time and resource. My requirement is to upload file form local database and use some mechanisms to check the file is complete.

svijay-MSFT 5,201 Reputation points Microsoft Employee

2021-08-23T14:44:35.13+00:00

Hello @Teddy ,

Welcome to the Microsoft Q&A platform.

With regards to your Question 1 - Can you please let me know which activity is this query regard to like Copy Activity etc. ?

To answer your second question - Yes - if the activities are set to run in Parallel.

A Sample with 3 Wait Activity running parallelly

If you closely look at the - Start time is same - indicating the all the activities were started around the same time.

Query 3: Are you planning to run 32 Copy activities? or are you referring to the Maximum concurrent Connection property of the Copy Activity? An additional context will help me understand this further.

Query 4: Are you looking for Move Activity rather than performing Copy Activity and Delete Activity. If that's the case - Unfortunately, this is not available. However, I'll check on this further.
svijay-MSFT 5,201 Reputation points Microsoft Employee

2021-08-30T03:52:12.327+00:00

@Teddy - Just checking in if you have had a chance to see the previous response. Please let us know with the information requested to understand further.