Performance of ADF pipeline after running for more than 12hours

Question

Hi Team,

It is observed that performance of the data factory pipeline that is using Azure IR to copy data from a ADLS Gen2 Storage account to another ADLS Gen2 Storage account is slowing down after 12hours. When restarted, the speed is picking up. Please share any suggestions to ensure consistent performance or improve the performance in such scenarios.

Pipeline Run ID - 1fa60a85-4349-4f12-9884-92ece1db6fb8

It took ~3hours to copy 4TB of data as below (Throughput - 423MBPS)

But other activity which started after this is completed is taking more time as below (Throughput - 35 mbps, very less compared to above one that is causing it to run for longer duration)

Thanks,
Haritha

Answer 1

Hello @Haritha Maddi and welcome to Microsoft Q&A.

From what I can see in your screenshots, the setting are all the same. Resources used are approximately the same. Yet transfer time is much too different. Obviously the settings are not the cause.

I checked for service disruptions, and none I could find in that time frame were in West Europe. So, I do not think service disruption to be the cause.

IThere are two possibilities left that I can think of. 1) Another service was making heavy use of your storage account at the same time, competing for I/O. 2) The integration runtime was unhealthy.

Given your report that the speed was fixed after cancelling and starting a new run, the second option feels more likely. I'll look through my resources, but I may have to pass this off to a support ticket.