How to automate indexing additional documents in Azure Open AI to access GPT model (gptv4 Turbo) and Azure Search Service for data indexing and retrieval.

Narendra Mahendrakar (Microland) 0 Reputation points
2024-04-10T10:57:42.9033333+00:00

I've created a PoC using https://github.com/Azure-Samples/azure-search-openai-demo and explored the ChatGPT like experience over enterprise data using RAG model which uses Azure OpenAI to access GPT model (gptv4 Turbo) and Azure Search Service for data indexing and retrieval.

It works well and i can interact with my enterprise data from an Azure WebApp which acts as a front-end chat bot.

For any new additional documents, i upload them to /Data folder from my Git codespace and run prepdocs.ps1 or prepdocs.sh script which will index additional documents and upload to Azure blob storage and i can interact with chat bot based on new document content.

Now i want to automate the process of indexing additional documents whenever any new document is being added to any repository, ex: SharePoint folder or any Azure Blob storage, the automation workflow should pick the new document, index and upload to Azure blob storage.

I have tried using Azure Automation Account runbooks to run prepdocs.ps1 script but, the script is dependant on so many other scripts due to which i've not been able to use it.

I also tried using LogicApps to trigger whenever any new documetn is added to SharePoint or Azure blob storage and action the Indexing the document and upload to blob storage, however this doesn't work too.

Can you please help me a better and easy way of automating process of indexing additonal documents and uploading to Azure blob storage? Thank you!

Azure AI Search
Azure AI Search
An Azure search service with built-in artificial intelligence capabilities that enrich information to help identify and explore relevant content at scale.
727 questions
Azure Logic Apps
Azure Logic Apps
An Azure service that automates the access and use of data across clouds without writing code.
2,867 questions
Azure Automation
Azure Automation
An Azure service that is used to automate, configure, and install updates across hybrid environments.
1,131 questions
Azure OpenAI Service
Azure OpenAI Service
An Azure service that provides access to OpenAI’s GPT-3 models with enterprise capabilities.
2,214 questions
Azure AI Document Intelligence
Azure AI Document Intelligence
An Azure service that turns documents into usable data. Previously known as Azure Form Recognizer.
1,403 questions
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. ajkuma 22,521 Reputation points Microsoft Employee
    2024-04-12T11:31:25.4333333+00:00

    @Narendra Mahendrakar (Microland) , Based on my understanding of your requirement, if you haven't checked, kindly checkout the "Chat with your data" solution accelerator is a GitHub repository offering a template for end-to-end RAG pattern implementation using the "Retrieve-then-read" interaction pattern.

    The repository includes instructions on customization and usage to suit your specific needs. This solution accelerator combines the GPT model in Azure OpenAI Service with an Azure AI Search index generated from your data, integrating them into a web application to deliver a natural language interface for search queries. Kindly checkout this article: Introducing the “Chat with your data” solution accelerator – now available on GitHub

    0 comments