Hello, i have a question regarding loading data to Cosmos DB database.
In our scenario, we had a PostgreSQL database and an app that exports three SQL views into CSV files. The total size of those CSV files is about 5 GB. Our colleague needs those files to perform some analytics in PowerBI app.
We generated those files every night and uploaded them into Azure Storage. Then, our colleague downloads those files on his computer and performs his work.
We wanted to use CosmosDB to load data from CSV files and then that data would automatically migrate do Synapse Analytics using Synapse Link.
However, i tried three ways of loading data into Cosmos DB and all of them seem to work very slowly.
First method - loading data using Azure Data Factory pipeline - it worked for about six hour per file.
Second method - i wrote an console app based on this code:
https://github.com/Azure-Samples/cosmos-dotnet-bulk-import-throughput-optimizer
I added reading data from my CSV file and run it, it worked for about six hour for one file.
Third method - i wrote a console app based on this code:
https://github.com/Azure/azure-cosmosdb-bulkexecutor-dotnet-getting-started
This one was the fastest, it loaded one CSV file in about 4 hours and 40 minutes, but it's still slow.
Is there any way to make it faster? Would you recommend us any other way of loading data into Cosmos DB, that would work faster?
PS our Cosmos DB account runs in serverless mode.