Most of the times expected data is not retrieved from Cognitive AI search and returns information is not available.

Question

I am trying to incorporate AzureCognitiveSearch for my GenAI use case. I have my files uploaded in Azure blob storage and have to retrieve key information(through intelligent search), summarization - chatgpt on our own data from multiple pdf documents and do comparison within the same or across multiple documents. I am using the basic search resource pricing tier for Cognitive search. But for most of the times post uploading the data and creating the index, the search result returns 'the requested information is not available in the retrieved data'.

Require assistance here as to:

1. Why it not able to retrieve the expected results for most cases even though appropriate data is present?
2. Refresh of the index has to be done automatically when file is uploaded/deleted from the blob.

Answer 1

@Kaustav Ghosh (LTIMINDTREE LIMITED) Thanks for your time over offline discussion. I am summarizing our offline conversation and posting it as answer to help other community members.

1. Why it not able to retrieve the expected results for most cases even though appropriate data is present?

Customer has investigated and found that, this issue has happened because they are querying using wrong search queries.

2. Refresh of the index has to be done automatically when file is uploaded/deleted from the blob.

Azure storage indexer supports change detection mechanism which keep track of the last update using the built-in time stamps on object and it will automatically add the newly added documents.

Although change detection is a given, deletion detection isn't. An indexer doesn't track object deletion in data sources. To avoid having orphan search documents, you can implement a "soft delete" strategy that results in deleting search documents first, with physical deletion in Azure Storage. Refer to this documentation for more information.

Hope this helps, let me know if you have any further questions on this.