Event Grid data connection
Event Grid ingestion is a pipeline that listens to Azure storage, and updates Azure Data Explorer to pull information when subscribed events occur. Azure Data Explorer offers continuous ingestion from Azure Storage (Blob storage and ADLSv2) with Azure Event Grid subscription for blob created or blob renamed notifications and streaming these notifications to Azure Data Explorer via an Azure Event Hubs.
The Event Grid ingestion pipeline goes through several steps. You create a target table in Azure Data Explorer into which the data in a particular format will be ingested. Then you create an Event Grid data connection in Azure Data Explorer. The Event Grid data connection needs to know events routing information, such as what table to send the data to and the table mapping. You also specify ingestion properties, which describe the data to be ingested, the target table, and the mapping. You can generate sample data and upload blobs or rename blobs to test your connection. Delete blobs after ingestion.
Event Grid ingestion can be managed through the Azure portal, using one-click ingestion, programmatically with C# or Python, or with the Azure Resource Manager template.
For general information about data ingestion in Azure Data Explorer, see Azure Data Explorer data ingestion overview.
Data format
- See supported formats.
- See supported compressions.
- The original uncompressed data size should be part of the blob metadata, or else Azure Data Explorer will estimate it. The ingestion uncompressed size limit per file is 4 GB.
Note
Event Grid notification subscription can be set on Azure Storage accounts for BlobStorage, StorageV2, or Data Lake Storage Gen2.
Ingestion properties
You can specify ingestion properties of the blob ingestion via the blob metadata. You can set the following properties:
| Property | Property description |
|---|---|
rawSizeBytes |
Size of the raw (uncompressed) data. For Avro/ORC/Parquet, that is the size before format-specific compression is applied. Provide the original data size by setting this property to the uncompressed data size in bytes. |
kustoTable |
Name of the existing target table. Overrides the Table set on the Data Connection blade. |
kustoDataFormat |
Data format. Overrides the Data format set on the Data Connection blade. |
kustoIngestionMappingReference |
Name of the existing ingestion mapping to be used. Overrides the Column mapping set on the Data Connection blade. |
kustoIgnoreFirstRecord |
If set to true, Kusto ignores the first row of the blob. Use in tabular format data (CSV, TSV, or similar) to ignore headers. |
kustoExtentTags |
String representing tags that will be attached to resulting extent. |
kustoCreationTime |
Overrides $IngestionTime for the blob, formatted as an ISO 8601 string. Use for backfilling. |
Events routing
When setting up a blob storage connection to Azure Data Explorer cluster, specify target table properties:
- table name
- data format
- mapping
This setup is the default routing for your data, sometimes referred to as static routing.
You can also specify target table properties for each blob, using blob metadata. The data will dynamically route, as specified by ingestion properties.
The following example shows you how to set ingestion properties on the blob metadata before uploading it. Blobs are routed to different tables.
For more information, see upload blobs.
// Blob is dynamically routed to table `Events`, ingested using `EventsMapping` data mapping
blob = container.GetBlockBlobReference(blobName2);
blob.Metadata.Add("rawSizeBytes", "4096"); // the uncompressed size is 4096 bytes
blob.Metadata.Add("kustoTable", "Events");
blob.Metadata.Add("kustoDataFormat", "json");
blob.Metadata.Add("kustoIngestionMappingReference", "EventsMapping");
blob.UploadFromFile(jsonCompressedLocalFileName);
Upload blobs
You can create a blob from a local file, set ingestion properties to the blob metadata, and upload it. For examples, see Ingest blobs into Azure Data Explorer by subscribing to Event Grid notifications
Note
- Use
BlockBlobto generate data.AppendBlobis not supported. - Using Azure Data Lake Gen2 storage SDK requires using
CreateFilefor uploading files andFlushat the end with the close parameter set to "true". For a detailed example of Data Lake Gen2 SDK correct usage, see upload file using Azure Data Lake SDK. - When the event hub endpoint doesn't acknowledge receipt of an event, Azure Event Grid activates a retry mechanism. If this retry delivery fails, Event Grid can deliver the undelivered events to a storage account using a process of dead-lettering. For more information, see Event Grid message delivery and retry.
Rename blobs
When using ADLSv2, you can rename a blob to trigger blob ingestion to Azure Data Explorer. For example, see Ingest blobs into Azure Data Explorer by subscribing to Event Grid notifications.
Note
- Directory renaming is possible in ADLSv2, but it doesn't trigger blob renamed events and ingestion of blobs inside the directory. To ingest blobs following renaming, directly rename the desired blobs.
- If you defined filters to track specific subjects while creating the data connection or while creating Event Grid resources manually, these filters are applied on the destination file path.
Delete blobs using storage lifecycle
Azure Data Explorer won't delete the blobs after ingestion. Use Azure Blob storage lifecycle to manage your blob deletion. It's recommended to keep the blobs for three to five days.
Known Event Grid issues
- When using Azure Data Explorer to export the files used for Event Grid ingestion, note:
- Event Grid notifications aren't triggered if the connection string provided to the export command or the connection string provided to an external table is a connecting string in ADLS Gen2 format (for example,
abfss://filesystem@accountname.dfs.core.windows.net) but the storage account isn't enabled for hierarchical namespace. - If the account isn't enabled for hierarchical namespace, connection string must use the Blob Storage format (for example,
https://accountname.blob.core.windows.net). The export works as expected even when using the ADLS Gen2 connection string, but notifications won't be triggered and Event Grid ingestion won't work.
- Event Grid notifications aren't triggered if the connection string provided to the export command or the connection string provided to an external table is a connecting string in ADLS Gen2 format (for example,
Next steps
- Ingest blobs into Azure Data Explorer by subscribing to Event Grid notifications
- Create an Event Grid data connection for Azure Data Explorer by using C#
- Create an Event Grid data connection for Azure Data Explorer by using Python
- Create an Event Grid data connection for Azure Data Explorer by using Azure Resource Manager template
- Use one-click ingestion to create an Event Hubs data connection for Azure Data Explorer
- Use one-click ingestion to ingest CSV data from a container to a new table in Azure Data Explorer