Azure Files
An Azure service that offers file shares in the cloud.
1,156 questions
Is there any solution/answer yet? This problem still continuous after 6 months.
Azure: azurefile shared pvc ReadWriteMany slow read and especially slow write speed
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(80, 8%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMJ%3C/text%3E%3C/svg%3E)
Marc Jansen
11
Reputation points
Hello there,
we're running a webserver in an AKS and the files for our cms (content management system) are located in a shared pvc (since the cms runs with a replica count of 3). The performance we get is bad. Not bearable bad.
The cms isn't huge, 7mb. Archived the transfer is almost instantly but when a lot of small files get transferred the speed drops dramatically. Same 7mb but it took almost 2 minutes on a 128gb azurefile-premium volume to transfer the files from the pod itself to the mounted pvc. The performance is also identically bad while the pvc is mounted to only 1 pod.
Is there any solution to this problem where azurefile storage performs really bad on a lot of small files?
Kind regards
{count} vote
-
Sumarigo-MSFT 43,401 Reputation points Microsoft Employee2021-09-24T14:34:27.387+00:00 @Marc Jansen May I know what speed are you getting(https://www.azurespeed.com/Azure/Latency) ?
For better understanding the issue Are you using Azure File storage "Premium"?Additional information: This article lists some common performance problems related to Azure file shares. It provides potential causes and workarounds for when you encounter these problems: https://learn.microsoft.com/en-us/azure/storage/files/storage-troubleshooting-files-performance
To transfer files you can use Azcopy (Fastest tools The tool works by maximizing utilization of the network link. It efficiently uses all available bandwidth, even over long-distance links. On a 10 Gbps link, it reaches around 4 TB per hour, which makes it about 3 to 10 times faster than competing tools we’ve tested. On slower links, Fast Data Transfer typically achieves over 90% of the link’s theoretical maximum, while other tools may achieve substantially less. )or Azure storage Explorer tools
Azure File performance shouldn't be affected by the size of the dataset.
You can see the perf and scale targets for Azure File here: https://learn.microsoft.com/en-us/azure/storage/files/storage-files-scale-targets#azure-file-sync-scale-targets. Note that limits which are marked as not a hard limit are tested limits, and limits that are hard limits are hard limits :)
-
Marc Jansen 11 Reputation points
2021-09-28T08:59:13.26+00:00 Hi @Sumarigo-MSFT , my average latency to my datacenter is 1 to 2 ms in average. But that's not the problem. The problem also occurs while pods/containers from within the cluster are reading and writing to the ReadWriteMany volume. Our CMS is 150mb and writing this to the volume takes 13 minutes and reading from the volume (copy from mounted azurefile volume to a local directory in pod) takes 6 minutes. I also tested this with a zone redundant azurefile volume and stopped my test after exceeding 20 minutes reading.
Sadly, azcopy is no solution to this problem.
I'll post IO benchmark later.
-
Sumarigo-MSFT 43,401 Reputation points Microsoft Employee
2021-09-29T06:19:20.017+00:00 @Marc Jansen Can you provide me the IO benchmark information?
-
Marc Jansen 11 Reputation points
2021-10-20T15:03:47.277+00:00 Sorry for my late answer. One of the tools I used is "ioping". I attached a textfile with the timings from the affected ReadWriteMany AzureFile, timings from the unaffected ReadWriteOnce AzureFile and timings from the pod/container.
I tested three different sizes. 400B, 4KB and 400KB. IOPing however does confirm the problem with the AzureFile ReadWriteMany volume. The timings from the the AzureFile ReadWriteOnce volume are great! Fast responding and high rate. The timings from the AzureFile ReadWriteMany (shared) volume are disastrous. Even worse for many small files (like in my case, Webserver). With this we can't have a high availability webserver with a shared volume.
Kind regards
Marc Jansen -
Marc Jansen 11 Reputation points
2021-10-20T15:05:58.91+00:00 Nevermind, microsoft won't let me upload the 30kb .txt.
Sign in to comment
2 answers
Sort by: Most helpful
-
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(73.60000000000001, 21%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EF%3C/text%3E%3C/svg%3E)
FG 6 Reputation points2022-03-02T09:44:40.52+00:00 We would also appreciate an answer to this issue.
We are also using kubernetes with a shared pvc but not utilizing anything related to azure.
The read write performance for all kinds of files on the pvc is bad. Reading files from the local pod system is fast.
Maybe this issue is related to how-to-improve-very-poor-file-read-performance-wit.html and the problem is not related azure but the file access from c# inside kubernetes.Our target framework is net5.0.
Any help would be great.
Tanks