question

HarithaMaddi-4180 avatar image
0 Votes"
HarithaMaddi-4180 asked HarithaMaddi-4180 commented

Performance of ADF pipeline after running for more than 12hours

Hi Team,

It is observed that performance of the data factory pipeline that is using Azure IR to copy data from a ADLS Gen2 Storage account to another ADLS Gen2 Storage account is slowing down after 12hours. When restarted, the speed is picking up. Please share any suggestions to ensure consistent performance or improve the performance in such scenarios.

Pipeline Run ID - 1fa60a85-4349-4f12-9884-92ece1db6fb8

It took ~3hours to copy 4TB of data as below (Throughput - 423MBPS)

132313-image.png

But other activity which started after this is completed is taking more time as below (Throughput - 35 mbps, very less compared to above one that is causing it to run for longer duration)

132302-image.png

Thanks,
Haritha

azure-data-factory
image.png (75.7 KiB)
image.png (79.2 KiB)
· 4
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @HarithaMaddi-4180 and welcome to Microsoft Q&A.

From what I can see in your screenshots, the setting are all the same. Resources used are approximately the same. Yet transfer time is much too different. Obviously the settings are not the cause.

I checked for service disruptions, and none I could find in that time frame were in West Europe. So, I do not think service disruption to be the cause.

IThere are two possibilities left that I can think of. 1) Another service was making heavy use of your storage account at the same time, competing for I/O. 2) The integration runtime was unhealthy.

Given your report that the speed was fixed after cancelling and starting a new run, the second option feels more likely. I'll look through my resources, but I may have to pass this off to a support ticket.

1 Vote 1 ·
HarithaMaddi-4180 avatar image HarithaMaddi-4180 MartinJaffer-MSFT ·

Thanks @MartinJaffer-MSFT for looking into the issue. Sure, we have raised a support ticket.

0 Votes 0 ·
MartinJaffer-MSFT avatar image MartinJaffer-MSFT HarithaMaddi-4180 ·

Looking into my resource, I found a few other scenarios, @HarithaMaddi-4180 .

In one case, customer had the number of files stay the same, but the number of folders/directories more than tripled. The increased amount of recursion slowed down their copy.
Due to the nearness in time your copies ran, I doubt this was the case.

Is your slow copy a 1-off, or does it happen repeatedly?
Is your copy slow at the same time every day, or is it random?

1 Vote 1 ·
Show more comments

0 Answers