Monitor Azure Data Explorer ingestion, commands, queries, and tables using diagnostic logs

Azure Data Explorer is a fast, fully managed data analytics service for real-time analysis on large volumes of data streaming from applications, websites, IoT devices, and more. Azure Monitor diagnostic logs provide data about the operation of Azure resources. Azure Data Explorer uses diagnostic logs for insights on ingestion, commands, query, and tables. You can export operation logs to Azure Storage, Event Hub, or Log Analytics to monitor ingestion, commands, and query status. Logs from Azure Storage and Azure Event Hub can be routed to a table in your Azure Data Explorer cluster for further analysis.

Important

Diagnostic log data may contain sensitive data. Restrict permissions of the logs destination according to your monitoring needs.

Prerequisites

Set up diagnostic logs for an Azure Data Explorer cluster

Diagnostic logs can be used to configure the collection of the following log data:

Note

Ingestion logs are supported for queued ingestion to the ingestion endpoint using SDKs, data connections, and connectors.

Ingestion logs aren't supported for streaming ingestion, direct ingestion to the engine, ingestion from query, or set-or-append commands.

Note

Failed ingestion logs are only reported for the final state of an ingest operation, unlike the Ingestion result metric, which is emitted for transient failures that are retried internally.

  • Successful ingestion operations: These logs have information about successfully completed ingestion operations.
  • Failed ingestion operations: These logs have detailed information about failed ingestion operations including error details.
  • Ingestion batching operations: These logs have detailed statistics of batches ready for ingestion (duration, batch size and blobs count).

The data is then archived into a Storage account, streamed to an Event Hub, or sent to Log Analytics, as per your specifications.

Enable diagnostic logs

Diagnostic logs are disabled by default. To enable diagnostic logs, do the following steps:

  1. In the Azure portal, select the Azure Data Explorer cluster resource that you want to monitor.

  2. Under Monitoring, select Diagnostic settings.

    Add diagnostics logs

  3. Select Add diagnostic setting.

  4. In the Diagnostic settings window:

    Configure diagnostics settings

    1. Enter a Diagnostic setting name.
    2. Select one or more targets: a Log Analytics workspace, a storage account, or an Event Hub.
    3. Select logs to be collected: SucceededIngestion, FailedIngestion, IngestionBatching, Command, or Query, TableUsageStatistics, or TableDetails.
    4. Select metrics to be collected (optional).
    5. Select Save to save the new diagnostic logs settings and metrics.

New settings will be set in a few minutes. Logs then appear in the configured archival target (Storage account, Event Hub, or Log Analytics).

Note

If you send logs to Log Analytics, the SucceededIngestion, FailedIngestion, IngestionBatching, Command, Query, TableUsageStatistics and TableDetails logs will be stored in Log Analytics tables named: SucceededIngestion, FailedIngestion, ADXIngestionBatching, ADXCommand, ADXQuery, ADXTableUsageStatistics and ADXTableDetails respectively.

Diagnostic logs schema

All Azure Monitor diagnostic logs share a common top-level schema. Azure Data Explorer has unique properties for their own events. All logs are stored in a JSON format.

Ingestion logs schema

Log JSON strings include elements listed in the following table:

Name Description
time Time of the report
resourceId Azure Resource Manager resource ID
operationName Name of the operation: 'MICROSOFT.KUSTO/CLUSTERS/INGEST/ACTION'
operationVersion Schema version: '1.0'
category Category of the operation. SucceededIngestion, FailedIngestion or IngestionBatching. Properties differ for successful operation, failed operation or batching operation.
properties Detailed information of the operation.

Successful ingestion operation log

Example:

{
    "time": "2019-05-27 07:55:05.3693628",
    "resourceId": "/SUBSCRIPTIONS/12534000-8109-4D84-83AD-576C0D5E1AAA/RESOURCEGROUPS/myResourceGroup/PROVIDERS/MICROSOFT.KUSTO/CLUSTERS/mycluster",
    "operationName": "MICROSOFT.KUSTO/CLUSTERS/INGEST/ACTION",
    "operationVersion": "1.0",
    "category": "SucceededIngestion",
    "properties":
    {
        "SucceededOn": "2019-05-27 07:55:05.3693628",
        "OperationId": "b446c48f-6e2f-4884-b723-92eb6dc99cc9",
        "Database": "Samples",
        "Table": "StormEvents",
        "IngestionSourceId": "66a2959e-80de-4952-975d-b65072fc571d",
        "IngestionSourcePath": "https://kustoingestionlogs.blob.core.windows.net/sampledata/events8347293.json",
        "RootActivityId": "d0bd5dd3-c564-4647-953e-05670e22a81d"
    }
}

Properties of a successful operation diagnostic log

Name Description
SucceededOn Time of ingestion completion
OperationId Azure Data Explorer ingestion operation ID
Database Name of the target database
Table Name of the target table
IngestionSourceId ID of the ingestion data source
IngestionSourcePath Path of the ingestion data source or blob URI
RootActivityId Activity ID

Failed ingestion operation log

Example:

{
    "time": "2019-05-27 08:57:05.4273524",
    "resourceId": "/SUBSCRIPTIONS/12534000-8109-4D84-83AD-576C0D5E1AAA/RESOURCEGROUPS/myResourceGroup/PROVIDERS/MICROSOFT.KUSTO/CLUSTERS/mycluster",
    "operationName": "MICROSOFT.KUSTO/CLUSTERS/INGEST/ACTION",
    "operationVersion": "1.0",
    "category": "FailedIngestion",
    "properties":
    {
        "failedOn": "2019-05-27 08:57:05.4273524",
        "operationId": "5956515d-9a48-4544-a514-cf4656fe7f95",
        "database": "Samples",
        "table": "StormEvents",
        "ingestionSourceId": "eee56f8c-2211-4ea4-93a6-be556e853e5f",
        "ingestionSourcePath": "https://kustoingestionlogs.blob.core.windows.net/sampledata/events5725592.json",
        "rootActivityId": "52134905-947a-4231-afaf-13d9b7b184d5",
        "details": "Permanent failure downloading blob. URI: ..., permanentReason: Download_SourceNotFound, DownloadFailedException: 'Could not find file ...'",
        "errorCode": "Download_SourceNotFound",
        "failureStatus": "Permanent",
        "originatesFromUpdatePolicy": false,
        "shouldRetry": false
    }
}

Properties of a failed operation diagnostic log

Name Description
FailedOn Time of ingestion completion
OperationId Azure Data Explorer ingestion operation ID
Database Name of the target database
Table Name of the target table
IngestionSourceId ID of the ingestion data source
IngestionSourcePath Path of the ingestion data source or blob URI
RootActivityId Activity ID
Details Detailed description of the failure and error message
ErrorCode Ingestion error code
FailureStatus Permanent or Transient. Retry of a transient failure may succeed.
OriginatesFromUpdatePolicy True if failure originates from an update policy
ShouldRetry True if retry may succeed

Ingestion batching operation log

Example:

{
  "resourceId": "/SUBSCRIPTIONS/12534000-8109-4D84-83AD-576C0D5E1AAA/RESOURCEGROUPS/myResourceGroup/PROVIDERS/MICROSOFT.KUSTO/CLUSTERS/mycluster",
  "time": "2021-04-18T19:19:57.0211782Z",
  "operationVersion": "1.0",
  "operationName": "MICROSOFT.KUSTO/CLUSTERS/INGESTIONBATCHING/ACTION",
  "category": "IngestionBatching",
  "correlationId": "2bb51038-c7dc-4ebd-9d7f-b34ece4cb735",
  "properties": {
    "Timestamp": "2021-04-18T19:19:57.0211782Z",
    "Database": "Samples",
    "Table": "StormEvents",
    "BatchingType": "Time",
    "SourceCreationTime": "2021-04-18T19:14:53.9543732Z",
    "BatchTimeSeconds": 302.1449075,
    "BatchSizeBytes": 3988,
    "DataSourcesInBatch": 2,
    "RootActivityId": "2bb51038-c7dc-4ebd-9d7f-b34ece4cb735"
  }
}

Properties of an ingestion batching operation diagnostic log

Name Description
Timestamp The time of the batching reporting
Database Name of the database holding the target table
Table Name of the target table into which the data is ingested
BatchingType Type of batching. If the batch reached batching time, the value will be Time. For data size or number of files, the value will be Size. If the system had to flush the data, for example due to cluster scaling or internal reset of system components, the value will be SystemFlush.
SourceCreationTime Minimal time (UTC) at which blobs in this batch were created
BatchTimeSeconds Total batching time of this batch (seconds)
BatchSizeBytes Total uncompressed size of data in this batch (bytes)
DataSourcesInBatch Number of data sources in this batch
RootActivityId The operation's activity ID

Next steps