How to improve (very poor) file read performance with C# Core on an Azure Kubernetes PVC
After having done a number of tests, I've narrowed the performance problem down to a combination of C# and PVC.
Problem: Read a large(ish) file (140MB) using C# Core from a PVC. Here's the (simple) test program:
using System;
using System.IO;
namespace cmaApp
{
class Program
{
private static void readStream(String filePath) {
Console.WriteLine($"Reading {filePath}");
using (var sr = new StreamReader(new BufferedStream(new FileStream(filePath, FileMode.Open))))
{
int len = sr.ReadToEnd().Length;
Console.WriteLine($"dat len={len}");
}
}
private static void copyFile(String filePath) {
File.Copy(filePath, "./t2.dat");
}
static void Main(string[] args)
{
Console.WriteLine("Start!");
String filePath = "/uploads/3864.dat"; // PVC file
//String filePath = "./3864.dat"; // Local file
readStream(filePath);
//copyFile(filePath);
Console.WriteLine("Done!");
}
}
}
Here's what I did, on the command line in the pod:
Test 1. Read the file with C# directly from the PVC (takes 2 mins!!):
/cmaApp# time dotnet run
Start!
Reading /uploads/3864.dat
dat len=153508096
Done!
real 2m2.172s
user 0m3.437s
sys 0m3.357s
Test 2. Copy the file to a local folder in the pod and read with C# (take 3 seconds):
cmaApp# time dotnet run
Start!
Reading ./3864.dat
dat len=153508096
Done!
real 0m2.765s
user 0m2.910s
sys 0m0.762s
Test 3. Read the file with cat directly from the PVC (takes 4 secs):
cmaApp# time cat /uploads/3864.dat |wc
144187 5543913 153508096
real 0m4.615s
user 0m0.755s
sys 0m0.293s
Test 4. Copy the file to a local folder using cp (takes 4 secs):
cmaApp# time cp /uploads/3864.dat .
real 0m4.358s
user 0m0.000s
sys 0m0.213s
Test 5. Copy the file to a local folder using C# File.Copy (takes 13 secs):
cmaApp# time dotnet run
Start!
Done!
real 0m13.042s
user 0m2.687s
sys 0m0.913s
Seeing that the linux level cp copies the file fast, it's not a throttling / network issue.
Why is C# Core so slow? What can be done to improve it?