AI enrichment in Azure Cognitive Search
In Azure Cognitive Search, AI enrichment refers to built-in cognitive skills and custom skills that add analysis, transformations, and content generation during indexing. Enrichments create new information where none previously existed: extracting information from images, detecting sentiment, key phrases, and entities from text, to name a few. Enrichments also add structure to undifferentiated text. All of these processes result in making previously unsearchable content available to full text search scenarios. In many instances, enriched documents are useful for scenarios other than search, such as for knowledge mining.
Enrichment is defined by a skillset that's attached to an indexer. The indexer will extract and set up the content, while the skillset identifies, analyzes, and creates new information and structures from images, blobs, and other unstructured data sources. The output of an enrichment pipeline is either a search index or a knowledge store.
A skillset can contain built-in skills from Cognitive Search or embed external processing that you provide in a custom skill. Examples of a custom skill might be a custom entity module or document classifier targeting a specific domain such as finance, scientific publications, or medicine.
Built-in skills fall into these categories:
Natural language processing skills include entity recognition, language detection, key phrase extraction, text manipulation, sentiment detection (including opinion mining), and PII detection. With these skills, unstructured text is mapped as searchable and filterable fields in an index.
Image processing skills include Optical Character Recognition (OCR) and identification of visual features, such as facial detection, image interpretation, image recognition (famous people and landmarks) or attributes like image orientation. These skills create text representations of image content, making it searchable using the query capabilities of Azure Cognitive Search.
Built-in skills in Azure Cognitive Search are based on pre-trained machine learning models in Cognitive Services APIs: Computer Vision and Text Analytics. You can attach a Cognitive Services resource if you want to leverage these resources during content processing.
Natural language and image processing is applied during the data ingestion phase, with results becoming part of a document's composition in a searchable index in Azure Cognitive Search. Data is sourced as an Azure data set and then pushed through an indexing pipeline using whichever built-in skills you need.
Feature availability
AI enrichment is available in regions where Azure Cognitive Services is also available. You can check the current availability of AI enrichment on the Azure products available by region page. AI enrichment is available in all supported regions except:
- Australia Southeast
- China North 2
- Norway East
- Germany West Central
If your search service is located in one of these regions, you will not be able to create and use skillsets, but all other search service functionality is available and fully supported.
When to use AI enrichment
You should consider enrichment if your raw content is unstructured text, image content, or content that needs language detection and translation. Applying AI through the built-in cognitive skills can unlock this content, increasing its value and utility in your search and data science apps.
Additionally, you might consider adding a custom skill if you have open-source, third-party, or first-party code that you'd like to integrate into the pipeline. Classification models that identify salient characteristics of various document types fall into this category, but any package that adds value to your content could be used.
Use-cases for built-in skills
A skillset that's assembled using built-in skills is well-suited for the following application scenarios:
Optical Character Recognition (OCR) that recognizes typeface and handwritten text in scanned documents (JPEG) is perhaps the most commonly used skill. Attaching the OCR skill will identify, extract, and ingest text from JPEG files.
Text translation of multilingual content is another commonly used skill. Language detection is built into Text Translation, but you can also run Language Detection independently if you just want the language codes of the content in your corpus.
PDFs with combined image and text. Text in PDFs can be extracted during indexing without the use of enrichment steps, but the addition of image and natural language processing can often produce a better outcome than a standard indexing provides.
Unstructured or semi-structured documents containing content that has inherent meaning or context that is hidden in the larger document.
Blobs in particular often contain a large body of content that is packed into a single "field". By attaching image and natural language processing skills to an indexer, you can create new information that is extant in the raw content, but not otherwise surfaced as distinct fields. Some ready-to-use built-in cognitive skills that can help: Key Phrase Extraction and Entity Recognition (people, organizations, and locations to name a few).
Additionally, built-in skills can also be used restructure content through text split, merge, and shape operations.
Use-cases for custom skills
Custom skills can support more complex scenarios, such as recognizing forms, or custom entity detection using a model that you provide and wrap in the custom skill web interface. Several examples of custom skills include Forms Recognizer, integration of the Bing Entity Search API, and custom entity recognition.
Enrichment steps
An enrichment pipeline consists of indexers that have skillsets. A skillset defines the enrichment steps, and the indexer drives the skillset. When configuring an indexer, you can include properties like output field mappings that send enriched content to a search index or projections that define data structures in a knowledge store.
Post-indexing, you can access content via search requests through all query types supported by Azure Cognitive Search.
Step 1: Connection and document cracking phase
Indexers connect to external sources using information provided in an indexer data source. When the indexer connects to the resource, it will "crack documents" to extract text and images. Image content can be routed to skills that perform image processing, while text content is queued for text processing.
This step assembles all of the initial or raw content that will undergo AI enrichment. For each document, an enrichment tree is created. Initially, the tree is just a root node representation, but it will grow and gain structure during skillset execution.
Step 2: Skillset enrichment phase
A skillset defines the atomic operations that are performed on each document. For example, for text and images extracted from a PDF, a skillset might apply entity recognition, language detection, or key phrase extraction to produce new fields in your index that are not available natively in the source.
skillset can be minimal or highly complex, and determines not only the type of processing, but also the order of operations. Most skillsets contain about three to five skills.
A skillset, plus the output field mappings defined as part of an indexer, fully specifies the enrichment pipeline. For more information about pulling all of these pieces together, see Define a skillset.
Internally, the pipeline generates a collection of enriched documents. You can decide which parts of the enriched documents should be mapped to indexable fields in your search index. For example, if you applied the key phrase extraction and the entity recognition skills, those new fields would become part of the enriched document, and can be mapped to fields on your index. See Annotations to learn more about input/output formations.
Step 3: Indexing
Indexing is the process wherein raw and enriched content is ingested as fields in a search index, and as projections if you are also creating a knowledge store. The same enriched content can appear in both, using implicit or explicit field mappings to send the content to the correct fields.
Enriched content is generated during skillset execution, and is temporary unless you save it. In order for enriched content to appear in a search index, the indexer must have mapping information so that it can send enriched content to a field in a search index. Output field mappings set up these associations.
Saving enriched output
In Azure Cognitive Search, an indexer saves the output it creates.
A searchable index is one of the outputs that is always created by an indexer. Specification of an index is an indexer requirement, and when you attach a skillset, the output of the skillset, plus any fields that are mapped directly from the source, are used to populate the index. Usually, the outputs of specific skills, such as key phrases or sentiment scores, are ingested into the index in fields created for that purpose.
A knowledge store is an optional output, used for downstream apps like knowledge mining. A knowledge store is defined within a skillset. Its definition determines whether your enriched documents are projected as tables or objects (files or blobs). Tabular projections are well suited for interactive analysis in tools like Power BI, whereas files and blobs are typically used in data science or similar processes.
Finally, an indexer can cache enriched documents in Azure Blob Storage for potential reuse in subsequent skillset executions. The cache is for internal use. Cached enrichments are consumable by the same skillset that you rerun at a later date. Caching is helpful if your skillset include image analysis or OCR, and you want to avoid the time and expense of reprocessing image files.
Indexes and knowledge stores are fully independent of each other. While you must attach an index to satisfy indexer requirements, if your sole objective is a knowledge store, you can ignore the index after it's populated. Avoid deleting it though. If you want to rerun the indexer and skillset, you'll need the index in order for the indexer to run.
Using enriched content
When processing is finished, you have a search index consisting of enriched documents, fully text-searchable in Azure Cognitive Search. Querying the index is how developers and users access the enriched content generated by the pipeline. The index is like any other you might create for Azure Cognitive Search: you can supplement text analysis with custom analyzers, invoke fuzzy search queries, add filters, or experiment with scoring profiles to tune search relevance.
You might also have a knowledge store. The knowledge store contains data that can be consumed in knowledge mining scenarios like analytics or machine learning. You can use Storage Browser, Power BI, or any app that connects to Azure Storage.
Checklist: A typical workflow
When beginning a project, it's helpful to work with a subset of data. Indexer and skillset design is an iterative process, and you'll iterate more quickly if you're working with a small representative data set.
Create a data source that specifies a connection to your data.
Create a skillset to add enrichment.
Create an index schema that defines a search index.
Create an indexer to bring all of the above components together. Creating or running indexer retrieves the data, runs the skillset, and loads the index.
Run queries to evaluate results and modify code to update skillsets, schema, or indexer configuration.
To iterate over the above steps, reset the indexer before rebuilding the pipeline, or delete and recreate the objects on each run (recommended if you are using the free tier). You should also enable enrichment caching to reuse existing enrichments wherever possible.