Quickstart: Create an Azure Cognitive Search knowledge store in the Azure portal

Important

Knowledge store is currently in public preview. Preview functionality is provided without a service level agreement, and is not recommended for production workloads. For more information, see Supplemental Terms of Use for Microsoft Azure Previews.

Knowledge store is a feature of Azure Cognitive Search that persists output from a cognitive skills pipeline for subsequent analyses or downstream processing.

A pipeline accepts images and unstructured text as raw content, applies AI through Cognitive Services (such as image and natural language processing), and creates enriched content (new structures and information) as output. One of the physical artifacts created by a pipeline is a knowledge store, which you can access through tools to analyze and explore content.

In this quickstart, you'll combine services and data in the Azure cloud to create a knowledge store. Once everything is in place, you'll run the Import data wizard in the portal to pull it all together. The end result is original plus AI-generated content that you can view in the portal (Storage explorer).

If you don't have an Azure subscription, create a free account before you begin.

Create services and load data

This quickstart uses Azure Cognitive Search, Azure Blob storage, and Azure Cognitive Services for the AI.

Because the workload is so small, Cognitive Services is tapped behind the scenes to provide free processing for up to 20 transactions daily when invoked from Azure Cognitive Search. As long as you use the sample data we provide, you can skip creating or attaching a Cognitive Services resource.

  1. Download HotelReviews_Free.csv. This data is hotel review data saved in a CSV file (originates from Kaggle.com) and contains 19 pieces of customer feedback about a single hotel.

  2. Create an Azure storage account or find an existing account under your current subscription. You'll use Azure storage for both the raw content to be imported, and the knowledge store that is the end result.

    There are two requirements for this account:

    • Choose the same region as Azure Cognitive Search.

    • Choose the StorageV2 (general purpose V2) account type.

  3. Open the Blob services pages and create a container.

  4. Click Upload.

    Upload the data

  5. Select the HotelReviews-Free.csv file you downloaded in the first step.

    Create the Azure Blob container

  1. Create an Azure Cognitive Search service or find an existing service under the same subscription. You can use a free service for this quickstart.

You are now ready to move on the Import data wizard.

Run the Import data wizard

In the search service Overview page, click Import data on the command bar to create a knowledge store in four steps.

Import data command

Step 1: Create a data source

  1. In Connect to your data, choose Azure Blob storage, select the account and container you created.

  2. For the Name, enter hotel-reviews-ds.

  3. For Parsing mode, select Delimited text, and then select the First Line Contains Header checkbox. Make sure the Delimiter character is a comma (,).

  4. Enter your storage service Connection String that you saved in a previous step.

  5. For Container name, enter hotel-reviews.

  6. Click Next: Add AI enrichment (Optional).

    Create a data source object

  7. Continue to the next page.

Step 2: Add cognitive skills

In this wizard step, you will create a skillset with cognitive skill enrichments. The skills we use in this sample will extract key phrases and detect the language and sentiment. In a later step, these enrichments will be "projected" into a knowledge store as Azure tables.

  1. Expand Attach Cognitive Services. Free (Limited enrichments) is selected by default. You can use this resource because number of records in HotelReviews-Free.csv is 19 and this free resource allows up to 20 transactions a day.

  2. Expand Add cognitive skills.

  3. For Skillset name, enter hotel-reviews-ss.

  4. For Source data field, select reviews_text.

  5. For Enrichment granularity level, select Pages (5000 characters chunks)

  6. Select these cognitive skills:

    • Extract key phrases

    • Detect language

    • Detect sentiment

      Create a skillset

  7. Expand Save enrichments to knowledge store.

  8. Enter the Storage account Connection String that you saved in a previous step.

  9. Select these Azure table projections:

    • Documents
    • Pages
    • Key phrases

    Configure knowledge store

  10. Continue to the next page.

Step 3: Configure the index

In this wizard step, you will configure an index for optional full-text search queries. The wizard will sample your data source to infer fields and data types. You only need to select the attributes for your desired behavior. For example, the Retrievable attribute will allow the search service to return a field value while the Searchable will enable full text search on the field.

  1. For Index name, enter hotel-reviews-idx.

  2. For attributes, make these selections:

    • Select Retrievable for all fields.
    • Select Filterable and Facetable for these fields: Sentiment, Language, Keyphrases
    • Select Searchable for these fields: city, name, reviews_text, language, Keyphrases

    Your index should look similar to the following image. Because the list is long, not all fields are visible in the image.

    Configure an index

  3. Continue to the next page.

Step 4: Configure the indexer

In this wizard step, you will configure an indexer that will pull together the data source, skillset, and the index you defined in the previous wizard steps.

  1. For Name, enter hotel-reviews-idxr.
  2. For Schedule, keep the default Once.
  3. Click Submit to run the indexer. Data extraction, indexing, application of cognitive skills all happen in this step.

Monitor status

Cognitive skill indexing takes longer to complete than typical text-based indexing. The wizard should open the Indexer list in the overview page so that you can track progress. For self-navigation, go to the Overview page and click Indexers.

In the Azure portal, you can also monitor the Notifications activity log for a clickable Azure Cognitive Search notification status link. Execution may take several minutes to complete.

Next steps

Now that you have enriched your data using Cognitive Services and projected the results into a knowledge store, you can use Storage Explorer or Power BI to explore your enriched data set.

You can view content in Storage Explorer, or take it a step further with Power BI to gain insights through visualization.

Tip

If you want to repeat this exercise or try a different AI enrichment walkthrough, delete the hotel-reviews-idxr indexer. Deleting the indexer resets the free daily transaction counter back to zero for Cognitive Services processing.