Manually create resources for Event Grid ingestion
Azure Data Explorer offers continuous ingestion from Azure Storage (Azure Blob storage and Azure Data Lake Storage Gen2) using an Event Grid Ingestion pipeline. In the Event Grid ingestion pipeline, an Azure Event Grid service routes blob created or blob renamed events from a storage account to Azure Data Explorer via an Azure Event Hubs.
In this article, you learn how to create manually the resources needed for Event Grid Ingestion: Event Grid subscription, event hub namespace, and event hub. event hub namespace and event hub creation are described in the Prerequisites. To use automatic creation of these resources while defining the Event Grid ingestion, see Create an Event Grid data connection in Azure Data Explorer.
Prerequisites
- An Azure subscription. Create a free Azure account.
- Create a cluster and database.
- A storage account.
- Event Grid notification subscription can be set on Azure Storage accounts for
BlobStorage,StorageV2, or Data Lake Storage Gen2. - An event hub namespace and event hub.
Note
For best performance, create all resources in the same region as the Azure Data Explorer cluster.
Create an Event Grid subscription
In the Azure portal, go to your storage account.
In the left menu, select Events > Event Subscription.
In the Create Event Subscription window within the Basic tab, provide the following values:
Setting Suggested value Field description Name test-grid-connection The name of the event grid subscription that you want to create. Event Schema Event Grid Schema The schema that should be used for the Event Grid. Topic Type Storage account The type of event grid topic. Automatically populated. Source Resource gridteststorage1 The name of your storage account. Automatically populated. System Topic Name gridteststorage1... The system topic where Azure Storage publishes events. This system topic then forwards the event to a subscriber that receives and processes events. Automatically populated. Filter to Event Types Blob Created Which specific events to get notified for. When creating the subscription, select one of the supported types: Microsoft.Storage.BlobCreated or Microsoft.Storage.BlobRenamed. Blob renaming is supported only for ADLSv2 storage. In ENDPOINT DETAILS, select Event Hubs.
Click Select an endpoint and fill in the event hub you created, for example test-hub.
Select the Filters tab if you want to track specific subjects. Set the filters for the notifications as follows:
- Select Enable subject filtering
- Subject Begins With field is the literal prefix of the subject. Since the pattern applied is startswith, it can span multiple containers, folders, or blobs. No wildcards are allowed.
- To define a filter on the blob container, set the field as follows:
/blobServices/default/containers/[container prefix]. - To define a filter on a blob prefix (or a folder in Azure Data Lake Gen2), set the field as follows:
/blobServices/default/containers/[container name]/blobs/[folder/blob prefix].
- To define a filter on the blob container, set the field as follows:
- Subject Ends With field is the literal suffix of the blob. No wildcards are allowed.
- Case-sensitive subject matching field indicates whether the prefix and suffix filters are case-sensitive.
For more information about filtering events, see blob storage events.
Select Create
Next steps
Continue the setup and create a data ingestion connection to Azure Data Explorer via Azure portal: Create an Event Grid data connection in Azure Data Explorer.
If you don't plan to continue Event Grid ingestion using the resources you created and don't want to use the resources anymore, clean up resources to avoid incurring costs.