I am loading data from csv files (file size ~40 MB) to Azure Cosmos DB using Azure Data Factory.If data mismatches happened during dataload, I want to delete the data entered using the file and reload once again.The data will be loaded across multiple partitions.I tried deleting data using Azure Functions,but will get timed out 230sec.
Please help me with the preferred way to delete large data from multiple partitions. Kindly share any links that explains the method.
Does your function calls Cosmos DB sproc from your Azure Functions to delete data?
No.I am using container.delete_item method to delete each item. If the csv has 30,000 records we need to make 30,000 calls as the partition key will be different for items in file
Microsoft team/Experts,
Please throw some insights on the optimal way to delete data loaded by files with sizes of 30-50 MB span across different partitions as explained in the query .I am getting stuck here.
It depends on how many partitions we are talking about.
You can create a stored procedure that takes a partionKey and then queries all items in that partition and deletes them. Then you don't have get all the items in the Azure Functions first.
7 people are following this question.
