Configure streaming ingestion on your Azure Data Explorer cluster using the Azure portal
Use streaming ingestion to load data when you need low latency between ingestion and query. The streaming ingestion operation completes in under 10 seconds, and your data is immediately available for query after completion. This ingestion method is suitable for ingesting a high volume of data, such as thousands of records per second, spread over thousands of tables. Each table receives a relatively low volume of data, such as a few records per second.
Use bulk ingestion instead of streaming ingestion when the amount of data ingested exceeds 4 GB per hour per table.
To learn more about different ingestion methods, see data ingestion overview.
- If you don't have an Azure subscription, create a free Azure account before you begin.
- Create an Azure Data Explorer cluster and database
Enable streaming ingestion on your cluster
Enable streaming ingestion while creating a new cluster in the Azure portal
You can enable streaming ingestion while creating a new Azure Data Explorer cluster.
In the Configurations tab, select Streaming ingestion > On.
Enable streaming ingestion on an existing cluster in the Azure portal
In the Azure portal, go to your Azure Data Explorer cluster.
In Settings, select Configurations.
In the Configurations pane, select On to enable Streaming ingestion.
Review the limitations prior to enabling steaming ingestion.
Create a target table and define the policy in the Azure portal
In the Azure portal, navigate to your cluster.
To create the table that will receive the data via streaming ingestion, copy the following command into the Query pane and select Run.
.create table TestTable (TimeStamp: datetime, Name: string, Metric: int, Source:string)
Define the streaming ingestion policy on the table you've created or on the database that contains this table.
A policy that is defined at the database level applies to all existing and future tables in the database.
Copy one of the following commands into the Query pane and select Run.
.alter table TestTable policy streamingingestion enable
.alter database StreamingTestDb policy streamingingestion enable
Use streaming ingestion to ingest data to your cluster
Two streaming ingestion types are supported:
- Event Hub or IoT Hub, which is used as a data source.
- Custom ingestion requires you to write an application that uses one of the Azure Data Explorer client libraries. See streaming ingestion sample for a sample application.
Choose the appropriate streaming ingestion type
|Criterion||Event Hub||Custom Ingestion|
|Data delay between ingestion initiation and the data available for query||Longer delay||Shorter delay|
|Development overhead||Fast and easy setup, no development overhead||High development overhead for application to handle errors and ensure data consistency|
Disable streaming ingestion on your cluster
Disabling streaming ingestion may take a few hours.
Before disabling streaming ingestion on your Azure Data Explorer cluster, drop the streaming ingestion policy from all relevant tables and databases. The removal of the streaming ingestion policy triggers data rearrangement inside your Azure Data Explorer cluster. The streaming ingestion data is moved from the initial storage to permanent storage in the column store (extents or shards). This process can take between a few seconds to a few hours, depending on the amount of data in the initial storage.
Drop the streaming ingestion policy in the Azure portal
In the Azure portal, go to your Azure Data Explorer cluster and select Query.
To drop the streaming ingestion policy from the table, copy the following command into Query pane and select Run.
.delete table TestTable policy streamingingestion
In Settings, select Configurations.
In the Configurations pane, select Off to disable Streaming ingestion.
- Database cursors aren't supported for a database if the database itself or any of its tables have the Streaming ingestion policy defined and enabled.
- Data mappings must be pre-created for use in streaming ingestion. Individual streaming ingestion requests don't accommodate inline data mappings.
- Streaming ingestion performance and capacity scales with increased VM and cluster sizes. The number of concurrent ingestion requests is limited to six per core. For example, for 16 core SKUs, such as D14 and L16, the maximal supported load is 96 concurrent ingestion requests. For two core SKUs, such as D11, the maximal supported load is 12 concurrent ingestion requests.
- The data size limit for streaming ingestion request is 4 MB.
- Schema updates, such as creation and modification of tables and ingestion mappings, may take up to five minutes for the streaming ingestion service. For more information see Streaming ingestion and schema changes.
- Enabling streaming ingestion on a cluster, even when data isn't ingested via streaming, uses part of the local SSD disk of the cluster machines for streaming ingestion data and reduces the storage available for hot cache.
- Extent tags can't be set on the streaming ingestion data.
- Update policy. The update policy can reference only the newly-ingested data in the source table and not any other data or tables in the database.
- If streaming ingestion is used on any of the tables of the database, this database cannot be used as leader for follower databases or as a data provider for Azure Data Share.