Configure streaming ingestion on your Azure Data Explorer cluster using the Azure portal

Use streaming ingestion to load data when you need low latency between ingestion and query. The streaming ingestion operation completes in under 10 seconds, and your data is immediately available for query after completion. This ingestion method is suitable for ingesting a high volume of data, such as thousands of records per second, spread over thousands of tables. Each table receives a relatively low volume of data, such as a few records per second.

Use bulk ingestion instead of streaming ingestion when the amount of data ingested exceeds 4 GB per hour per table.

To learn more about different ingestion methods, see data ingestion overview.

Prerequisites

Enable streaming ingestion on your cluster

Enable streaming ingestion while creating a new cluster in the Azure portal

You can enable streaming ingestion while creating a new Azure Data Explorer cluster.

In the Configurations tab, select Streaming ingestion > On.

Enable streaming ingestion while creating a cluster in Azure Data Explorer

Enable streaming ingestion on an existing cluster in the Azure portal

  1. In the Azure portal, go to your Azure Data Explorer cluster.

  2. In Settings, select Configurations.

  3. In the Configurations pane, select On to enable Streaming ingestion.

  4. Select Save.

    Turn on streaming ingestion in Azure Data Explorer

Warning

Review the limitations prior to enabling steaming ingestion.

Create a target table and define the policy in the Azure portal

  1. In the Azure portal, navigate to your cluster.

  2. Select Query.

    Select query in the Azure Data Explorer portal to enable streaming ingestion

  3. To create the table that will receive the data via streaming ingestion, copy the following command into the Query pane and select Run.

    .create table TestTable (TimeStamp: datetime, Name: string, Metric: int, Source:string)
    

    Create a table for streaming ingestion into Azure Data Explorer

  4. Define the streaming ingestion policy on the table you've created or on the database that contains this table.

    Tip

    A policy that is defined at the database level applies to all existing and future tables in the database.

  5. Copy one of the following commands into the Query pane and select Run.

    .alter table TestTable policy streamingingestion enable
    

    or

    .alter database StreamingTestDb policy streamingingestion enable
    

    Define the streaming ingestion policy in Azure Data Explorer

Use streaming ingestion to ingest data to your cluster

Two streaming ingestion types are supported:

Choose the appropriate streaming ingestion type

Criterion Event Hub Custom Ingestion
Data delay between ingestion initiation and the data available for query Longer delay Shorter delay
Development overhead Fast and easy setup, no development overhead High development overhead for application to handle errors and ensure data consistency

Disable streaming ingestion on your cluster

Warning

Disabling streaming ingestion may take a few hours.

Before disabling streaming ingestion on your Azure Data Explorer cluster, drop the streaming ingestion policy from all relevant tables and databases. The removal of the streaming ingestion policy triggers data rearrangement inside your Azure Data Explorer cluster. The streaming ingestion data is moved from the initial storage to permanent storage in the column store (extents or shards). This process can take between a few seconds to a few hours, depending on the amount of data in the initial storage.

Drop the streaming ingestion policy in the Azure portal

  1. In the Azure portal, go to your Azure Data Explorer cluster and select Query.

  2. To drop the streaming ingestion policy from the table, copy the following command into Query pane and select Run.

    .delete table TestTable policy streamingingestion 
    

    Delete streaming ingestion policy in Azure Data Explorer

  3. In Settings, select Configurations.

  4. In the Configurations pane, select Off to disable Streaming ingestion.

  5. Select Save.

    Turn off streaming ingestion in Azure Data Explorer

Limitations

  • Database cursors aren't supported for a database if the database itself or any of its tables have the Streaming ingestion policy defined and enabled.
  • Data mappings must be pre-created for use in streaming ingestion. Individual streaming ingestion requests don't accommodate inline data mappings.
  • Streaming ingestion performance and capacity scales with increased VM and cluster sizes. The number of concurrent ingestion requests is limited to six per core. For example, for 16 core SKUs, such as D14 and L16, the maximal supported load is 96 concurrent ingestion requests. For two core SKUs, such as D11, the maximal supported load is 12 concurrent ingestion requests.
  • The data size limit for streaming ingestion request is 4 MB.
  • Schema updates, such as creation and modification of tables and ingestion mappings, may take up to five minutes for the streaming ingestion service. For more information see Streaming ingestion and schema changes.
  • Enabling streaming ingestion on a cluster, even when data isn't ingested via streaming, uses part of the local SSD disk of the cluster machines for streaming ingestion data and reduces the storage available for hot cache.
  • Extent tags can't be set on the streaming ingestion data.
  • Update policy. The update policy can reference only the newly-ingested data in the source table and not any other data or tables in the database.
  • If streaming ingestion is used on any of the tables of the database, this database cannot be used as leader for follower databases or as a data provider for Azure Data Share.

Next steps