Azure Cognitive Search monitoring data reference
This article provides a reference of log and metric data collected to analyze the performance and availability of Azure Cognitive Search. See Monitoring Azure Cognitive Search for details on collecting and analyzing monitoring data for your search service.
This section lists the platform metrics collected for Azure Cognitive Search (Microsoft.Search/searchServices).
|DocumentsProcessedCount||Count||Total of the number of documents successfully processed in an indexing operation (either by an indexer or by pushing documents directly).|
|SearchLatency||Seconds||Average search latency for queries that execute on the search service.|
|SearchQueriesPerSecond||CountPerSecond||Average of the search queries per second (QPS) for the search service. It's common for queries to execute in milliseconds, so only queries that measure as seconds will appear in a metric like QPS. The minimum is the lowest value for search queries per second that was registered during that minute. The same applies to the maximum value. Average is the aggregate across the entire minute. For example, within one minute, you might have a pattern like this: one second of high load that is the maximum for SearchQueriesPerSecond, followed by 58 seconds of average load, and finally one second with only one query, which is the minimum.|
|SkillExecutionCount||Count||Total number of skill executions processed during an indexer operation.|
|ThrottledSearchQueriesPercentage||Percent||Average percentage of the search queries that were throttled from the total number of search queries that executed during a one-minute interval.|
For reference, see a list of all platform metrics supported in Azure Monitor.
Dimensions of a metric are name/value pairs that carry additional data to describe the metric value. Azure Cognitive Search has the following dimensions associated with its metrics that capture a count of documents or skills that were executed ("Document processed count" and "Skill execution invocation count").
|DataSourceName||A named data source connection used during indexer execution. Valid values are one of the supported data source types.|
|Failed||Indicates whether the instance failed.|
|IndexerName||Name of an indexer.|
|IndexName||Name of an index.|
|SkillsetName||Name of a skillset used during indexer execution.|
|SkillName||Name of a skill within a skillset.|
|SkillType||The @odata.type of the skill.|
For more information on what metric dimensions are, see Multi-dimensional metrics.
Resource logs are platform logs that provide insight into operations that were performed within an Azure resource. Resource logs are generated by the search service automatically, but are not collected by default. You must create a diagnostic setting to send resource logs to a Log Analytics workspace to use with Azure Monitor Logs, Azure Event Hubs to forward outside of Azure, or to Azure Storage for archiving.
This section identifies the type (or category) of resource logs you can collect for Azure Cognitive Search:
- Resource logs are grouped by type (or category). Azure Cognitive Search generates resource logs under the Operations category.
For reference, see a list of all resource logs category types supported in Azure Monitor.
Azure Monitor Logs tables
Azure Monitor Logs is a feature of Azure Monitor that collects and organizes log and performance data from monitored resources. If you configured a diagnostic setting for Log Analytics, you can query the Azure Monitor Logs tables for the resource logs generated by Azure Cognitive Search.
This section refers to all of the Azure Monitor Logs Kusto tables relevant to Azure Cognitive Search and available for query by Log Analytics and Metrics Explorer in the Azure portal.
|AzureActivity||Entries from the Azure Activity log that provide insight into control plane operations. Tasks invoked on the control plane, such as adding or removing replicas and partitions, will be represented through a "Get Admin Key" activity.|
|AzureDiagnostics||Logged query and indexing operations.|
|AzureMetrics||Metric data emitted by Azure Cognitive Search that measures health and performance.|
For a reference of all Azure Monitor Logs / Log Analytics tables, see the Azure Monitor Log Table Reference.
Azure Cognitive Search uses the Azure Diagnostics table to collect resource logs related to queries and indexing on your search service.
The following table lists common operations related to Azure Cognitive Search that may be created in the Azure Activity log.
|Get Admin Key||Any operation that requires administrative rights will be logged as a "Get Admin Key" operation.|
|Get Query Key||Any read-only operation against the documents collection of an index.|
|Regenerate Admin Key||A request to regenerate either the primary or secondary admin API key.|
Common entries include references to API keys - generic informational notifications like Get Admin Key and Get Query keys. These activities indicate requests that were made using the admin key (create or delete objects) or query key, but do not show the request itself. For information of this grain, you must configure diagnostic logging.
Alternatively, you might gain some insight through change history. In Azure portal, select the activity to open the detail page and then select "Change history" for information about the underlying operation.
For more information on the schema of Activity Log entries, see Activity Log schema.
The following schemas are in use by Azure Cognitive Search. If you are building queries or custom reports, the data structures that contain Azure Cognitive Search resource logs conform to the schema below.
For resource logs sent to blob storage, each blob has one root object called records containing an array of log objects. Each blob contains records for all the operations that took place during the same hour.
Resource log schema
The following example illustrates a resource log that includes common properties (TimeGenerated, Resource, Category, and so forth) and search-specific properties (OperationName and OperationVersion).
|Name||Type||Description and example|
|TimeGenerated||Datetime||Timestamp of the operation. For example:
|Resource||String||Resource ID. For example:
|Category||String||"OperationLogs". This value is a constant. OperationLogs is the only category used for resource logs.|
|OperationName||String||The name of the operation (see the full list of operations below). An example is
|OperationVersion||String||The api-version used on the request. For example:
|ResultType||String||"Success". Other possible values: Success or Failure|
|ResultSignature||Int||An HTTP result code. For example:
|DurationMS||Int||Duration of the operation in milliseconds.|
|Properties||Object||Object containing operation-specific data. See the following properties schema table.|
The properties below are specific to Azure Cognitive Search.
|Name||Type||Description and example|
|Description_s||String||The operation's endpoint. For example:
|Documents_d||Int||Number of documents processed.|
|IndexName_s||String||Name of the index associated with the operation.|
|Query_s||String||The query parameters used in the request. For example:
OperationName values (logged operations)
The operations below can appear in a resource log.
|DataSources.*||Applies to indexer data sources. Can be Create, Delete, Get, List.|
|DebugSessions.*||Applies to a debug session. Can be Create, Delete, Get, List, Start, and Status.|
|DebugSessions.DocumentStructure||An enriched document is loaded into a debug session.|
|DebugSessions.RetrieveIndexerExecutionHistoricalData||A request for indexer execution details.|
|DebugSessions.RetrieveProjectedIndexerExecutionHistoricalData||Execution history for enrichments projected to a knowledge store.|
|Indexers.*||Applies to an indexer. Can be Create, Delete, Get, List, and Status.|
|Indexes.*||Applies to a search index. Can be Create, Delete, Get, List.|
|indexes.Prototype||This is an index created by the Import Data wizard.|
|Indexing.Index||This operation is a call to Add, Update or Delete Documents.|
|Metadata.GetMetadata||A request for search service system data.|
|Query.Autocomplete||An autocomplete query against an index. See Query types and composition.|
|Query.Lookup||A lookup query against an index. See Query types and composition.|
|Query.Search||A full text search request against an index. See Query types and composition.|
|Query.Suggest||Type ahead query against an index. See Query types and composition.|
|ServiceStats||This operation is a routine call to Get Service Statistics, either called directly or implicitly to populate a portal overview page when it is loaded or refreshed.|
|Skillsets.*||Applies to a skillset. Can be Create, Delete, Get, List.|