question

PeterMa-0048 avatar image
0 Votes"
PeterMa-0048 asked SumanthMarigowda-MSFT edited

Very slow when writing lots of files to Blob Storage

I wrote about 1000 small files(~0.5 mb/file) to blob storage from ADF.
I only found 70 files by Azure sotrage explorer after ADF activity finished.

When I refresh, there will be 1 or 2 files added to the folder like the following image show.

My question is, are all files wrote to blob storage at that time when ADF activity finsihed. Because I have another activity to consume these files but I don't know when all the files can be writen to blob storage?

114347-image.png


azure-blob-storage
image.png (100.8 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

SumanthMarigowda-MSFT avatar image
0 Votes"
SumanthMarigowda-MSFT answered SumanthMarigowda-MSFT edited

anonymous user-0048 Welcome to Microsoft Q&A Forum, Thank you for posting your query here!

There could be several reason, for the performance issue (Network bandwidth, intermittently issue may occur). Can you perform
https://www.azurespeed.com/Azure/Download

Which version on Azure Storage explorer are using? Can you try to un-install and re-install the latest version of Storage Explorer.

For testing purpose you can also use azcopy tool and check the performance

Storage Explorer currently doesn't have the functionality for users to configure our usage of the network bandwidth. It is something we want to enable in the future as a part of the configuration. There are two workarounds I can think of in addition to using FTP. Storage Explorer has a built-in Experimental feature called AzCopy integration, which dramatically speeds up blob uploading and downloading. You can also use AzCopy at command line which actually gives you the ability to configure different parameters including the bandwidth you want to use during upload/download. Hope this is informational.

Additional information: There is no throttling on the Azure side below the published 60 MB/s scalability target for a single blob. If the network and client machine can handle the traffic then we will send it.

Here's a guide for troubleshooting high E2E latency: https://docs.microsoft.com/en-us/azure/storage/common/storage-monitoring-diagnosing-troubleshooting#metrics-show-high-AverageE2ELatency-and-low-AverageServerLatency


Kindly let us know if the above helps or you need further assistance on this issue.


Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.


· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi Sumanth,

The main issue here is when ADF to write the 1000 files too slow. I'm not care too much about to my local machine.
Are there any suggestions to improve the performance to speed it up?

Thanks,
Peter

0 Votes 0 ·

anonymous user-0048 This troubleshooting article should to optimize your performance issue: https://docs.microsoft.com/en-us/azure/data-factory/copy-activity-performance

When I check there is no drop in the availability ( there is client side latency compared to Server latency) , Azure Data Factory team can provide more insight to you on this scenario(refer to the above article). There is no issue with the Azure Storage account or Azure Storage Explorer, I would recommended to use azcopy tool which is the fastest tool for data transfer



Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

Screenshot has been shared offline


0 Votes 0 ·