Azure AI Search
An Azure search service with built-in artificial intelligence capabilities that enrich information to help identify and explore relevant content at scale.
767 questions
Azure OpenAI Chat RAG with Own Tabular Data not working at all
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(160, 54%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ER%3C/text%3E%3C/svg%3E)
rstudent07
0
Reputation points
I'm building an Azure AI Search + OpenAI chatbot to "talk/speak" with my own data. I'm in the sports field, and we need to create a chatbot capable of retrieving actual data from Excel files that we have and performing basic summary, sorting, and filtering tasks. Our data is mainly historical data on athletes. This is a quick look into what we have:
| Player | Scoring | Games-In | Red-Flags | Net-Worth | League |
|---|---|---|---|---|---|
| John Doe | 87 | 150 | 2 | 12,500,000 | Alpha League |
| Jane Smith | 92 | 200 | 1 | 25,000,000 | Beta League |
| Max Powers | 75 | 180 | 3 | 10,000,000 | Gamma League |
| Lucy Heart | 68 | 210 | 4 | 15,000,000 | Delta League |
| Jack Hunter | 80 | 170 | 2 | 8,000,000 | Epsilon League |
| Emma Stone | 95 | 220 | 1 | 30,000,000 | Zeta League |
| Liam Knight | 78 | 160 | 2 | 11,500,000 | Eta League |
| Ava Strong | 85 | 190 | 3 | 14,000,000 | Theta League |
| Noah Swift | 90 | 230 | 1 | 20,000,000 | Iota League |
| Mia Quick | 83 | 175 | 2 | 9,000,000 | Kappa League |
I have successfully created an index and semantic ranker with AI search, dividing our data into chunks to make it easier and more manageable for the search algorithm. We have also decided to include a file with the definitions of each feature. For instance, 'Net-Worth' is the value in US dollars of the athlete for business and financial purposes.
But when I ask my chatbot basic queries or questions like:
"What is the scoring of Jane Smith?" instead of retrieving 92, it says, "Sorry, I don't have sufficient data to answer that question."
When I ask it, "Who are the top 3 athletes with the highest net worth?" instead of giving a list sorted by player and their net worth, it just retrieves, "Sorry, I don't have sufficient data to answer that question."
Mind you, my data is extensive, with almost 15,000 entries which I have divided into very small chunks of information, keeping the headers to not lose context.
For those who have worked with tabular data with lots of quantitative and qualitative information in chatbots, what do you suggest is the best approach to solve these kinds of issues?
I'm currently dividing the Excel files into smaller Excel files, which are in XML format. Is it better to divide them into chunks of JSON, CSV, or .txt files? I would greatly appreciate any help from those of you who have worked with this type of data. Thank you
{count} votes
-
brtrach-MSFT 15,366 Reputation points Microsoft Employee2024-05-25T03:08:11.3633333+00:00 @rstudent07 One thing to consider is the format of the files you're using. While XML files can be a good choice for storing structured data, they can be more difficult to work with than other formats like CSV or JSON. CSV files, in particular, are a common choice for storing tabular data because they are easy to read and write and can be easily imported into a variety of tools and programming languages.
Another thing to consider is the way you're structuring your data. It's possible that the chatbot is having trouble understanding your queries because the data is not structured in a way that makes it easy to retrieve the information you're asking for. For example, if you're asking for the scoring of a particular player, but the data is organized by league, it may be difficult for the chatbot to find the information you're looking for.
To help your chatbot better understand your queries, you may want to consider using a natural language processing (NLP) tool like Azure Cognitive Services. NLP tools can help your chatbot understand the intent behind a user's query, even if the query is phrased in an unusual or unexpected way.
Also, it's possible that the chatbot simply doesn't have enough data to answer your queries accurately. While you may have a large amount of data, it's possible that the chatbot is having trouble finding the specific information it needs to answer your queries. In this case, you may want to consider adding more data to your index or refining your search algorithm to better match the queries you're receiving.
-
rstudent07 0 Reputation points
2024-05-25T13:25:03.66+00:00 @brtrach-MSFT Thank you for taking the time to answer I have already tried JSON CSV and text with no luck, it seems as though AI search will only look for a handful of text among the available and then retrieve a response my file has historical data of more than 5k rows and the chatbot cant complete the simplest sorting query :-(
Sign in to comment