I am getting the data from google big query using self hosted IR and save it into azure MySQL database using Azure Manged IR with VNET integration. Currently the transfer rate is slow. It took about an hour to transfer 1G of data.
I have read the official website about tuning the performance by adjusting the DIU and parallel copies. I have try to increase these parameters and the throughput does not really change much.
I have checked on azure MySQL database, the memory consumption is about 50% for 2vcore. I did try to increase to 4vcore and the memory consumption is reduced to about 25%. So I presume that it should not be the bottleneck for the process.
How should I identify the bottleneck of the process and improve the throughput?
Any advice would be greatly appreciated!