question

MarcJansen-9149 avatar image
0 Votes"
MarcJansen-9149 asked MarcJansen-9149 commented

Azure: azurefile shared pvc ReadWriteMany slow read and especially slow write speed

Hello there,

we're running a webserver in an AKS and the files for our cms (content management system) are located in a shared pvc (since the cms runs with a replica count of 3). The performance we get is bad. Not bearable bad.

The cms isn't huge, 7mb. Archived the transfer is almost instantly but when a lot of small files get transferred the speed drops dramatically. Same 7mb but it took almost 2 minutes on a 128gb azurefile-premium volume to transfer the files from the pod itself to the mounted pvc. The performance is also identically bad while the pvc is mounted to only 1 pod.

Is there any solution to this problem where azurefile storage performs really bad on a lot of small files?

Kind regards

azure-kubernetes-serviceazure-files
· 5
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@MarcJansen-9149 May I know what speed are you getting(https://www.azurespeed.com/Azure/Latency) ?
For better understanding the issue Are you using Azure File storage "Premium"?

Additional information: This article lists some common performance problems related to Azure file shares. It provides potential causes and workarounds for when you encounter these problems: https://docs.microsoft.com/en-us/azure/storage/files/storage-troubleshooting-files-performance

To transfer files you can use **Azcopy** (Fastest tools The tool works by maximizing utilization of the network link. It efficiently uses all available bandwidth, even over long-distance links. On a 10 Gbps link, it reaches around 4 TB per hour, which makes it about 3 to 10 times faster than competing tools we’ve tested. On slower links, Fast Data Transfer typically achieves over 90% of the link’s theoretical maximum, while other tools may achieve substantially less. )or Azure storage Explorer tools

Azure File performance shouldn't be affected by the size of the dataset.

You can see the perf and scale targets for Azure File here: https://docs.microsoft.com/en-us/azure/storage/files/storage-files-scale-targets#azure-file-sync-scale-targets. Note that limits which are marked as not a hard limit are tested limits, and limits that are hard limits are hard limits :)



0 Votes 0 ·

Hi @Sumarigo-MSFT , my average latency to my datacenter is 1 to 2 ms in average. But that's not the problem. The problem also occurs while pods/containers from within the cluster are reading and writing to the ReadWriteMany volume. Our CMS is 150mb and writing this to the volume takes 13 minutes and reading from the volume (copy from mounted azurefile volume to a local directory in pod) takes 6 minutes. I also tested this with a zone redundant azurefile volume and stopped my test after exceeding 20 minutes reading.

Sadly, azcopy is no solution to this problem.

I'll post IO benchmark later.

0 Votes 0 ·

@MarcJansen-9149 Can you provide me the IO benchmark information?

0 Votes 0 ·
Show more comments

0 Answers