Azure blob container - Improve performance in fetching pdf files from Azure blob container.

Question

Hi Team,

Below is my code to fetch the pdf files from Azure blob container. Most of the times the blob container contains up to 5000 pdf files.

Issue i am facing: Performance is very slow and taking more time to fetch when there are more than 500 pdf files in the blob container.

Need inputs and help to correct my code to improve the Performance. Expecting to see if there is any way to improve the code to fetch all the files in 5 seconds..

List<WorkflowDocumentListModel> documentListModel = new List<WorkflowDocumentListModel>();
if (request.RequestModel.WorkflowStatusId == (byte)WorkflowStatus.RCV) {
IList<PdfFileMetaDataResponse> RCVFileList = await _blobManager.GetAllFilesMetaDataAsync(UserContext.AzureBlobContainerWorkflowQueue, cancellationToken).ConfigureAwait(false);
foreach (PdfFileMetaDataResponse item in RCVFileList ) {
documentListModel.Add(new WorkflowDocumentListModel {
Id = 0,
Name = item.FileName,
AssignedTo = item.CustomUserName,
AssignedToId = item.CustomUserId,
AppDate = item.LastModified?.ClientTimeToUtc(UserContext.TimeZone),
IsGrabbedByUser = userId == item.CustomUserId
});
}

 public async Task<IList<PdfFileMetaDataResponse>> GetAllFilesMetaDataAsync(string containerName, CancellationToken cancellationToken)  
        {  
            BlobServiceClient _blobServiceClient = new BlobServiceClient(UserContext.AzureBlobStorageConnection);  
            BlobContainerClient containerClient = _blobServiceClient.GetBlobContainerClient(containerName);  
            List<PdfFileMetaDataResponse> responseList = new List<PdfFileMetaDataResponse>();  
            await foreach (BlobItem file in containerClient.GetBlobsAsync(cancellationToken: cancellationToken))  
            {  
                BlobClient blobClient = containerClient.GetBlobClient(file.Name);  
                BlobProperties blobProperties = await blobClient.GetPropertiesAsync(cancellationToken: cancellationToken).ConfigureAwait(false);  
  
                DateTime? customLastModifiedDate = ConvertWebApiFileLastModifiedMillisecondsToDateTime(  
                                                blobProperties.Metadata?.Where(i => i.Key.ToUpperInvariant() == CustomBlobMetadataLastModifiedMilliseconds.ToUpperInvariant())  
                                                                                .Select(i => i.Value)  
                                                                                .FirstOrDefault());  
                string customUserName = blobProperties.Metadata?.Where(i => i.Key.ToUpperInvariant() == CustomBlobMetadataUserName.ToUpperInvariant())  
                                                                                .Select(i => i.Value)  
                                                                                .FirstOrDefault();  
  
                bool hasCustomUserID = int.TryParse(blobProperties.Metadata?.Where(i => i.Key.ToUpperInvariant() == CustomBlobMetadataUserId.ToUpperInvariant())  
                                                                .Select(i => i.Value).FirstOrDefault(), out int customUserID);  
  
                PdfFileMetaDataResponse response = new PdfFileMetaDataResponse()  
                {  
                    FileName = file.Name,  
                    CustomUserName = customUserName,  
                    CustomUserId = hasCustomUserID ? customUserID : (int?)null,  
                    LastModified = customLastModifiedDate ?? file.Properties.LastModified?.UtcDateTime  
                };  
  
                responseList.Add(response);  
            }  
  
            return responseList;  
        }  

Thank you.

Answer 1

@Vamshi I see your question was answered on SO- https://stackoverflow.com/questions/73349120/azure-blob-container-improve-performance-in-fetching-pdf-files-and-files-detai

If you need any further assistance, please do let us know. Thank you!