Hi, I've set up an ADX Dev cluster which has been configured for streaming. I created a Database and Table which both had their streamingingestion policies enabled. I used this MS Link to create an ADX output from an Azure Stream Analytics job I currently have. I am getting the expected data from ASA into the ADX table. However, it's coming in at a rate of 5 minutes. When I looked at the Insights > Ingestion tab of the cluster, I realized that Ingestion Batching is being forced on my data despite the data coming in with under 2 sec stage latency. Is there any way to avoid the ingestion batching and use streaming ingestion for my ASA input?

Hello @Javier Huamani welcome to this moderated Azure community forum. The Azure Stream Analytics job output for Azure Data Explorer uses batching by default. According to the documentation it should be possible to enable streaming ingestion. Please check the documentation and especially check the steps in Configure streaming ingestion on your Azure Data Explorer cluster . For aggregation options, see IngestionBatching policy . If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. All community members with similar issues will benefit by doing so. Your contribution is highly appreciated.

How to achieve streaming ingestion for ASA input to ADX?

Accepted answer

Sander van de Velde 28,386 Reputation points MVP

2024-02-23T00:14:59.0466667+00:00

Hello @Javier Huamani

welcome to this moderated Azure community forum.

The Azure Stream Analytics job output for Azure Data Explorer uses batching by default.

According to the documentation it should be possible to enable streaming ingestion.

Please check the documentation and especially check the steps in Configure streaming ingestion on your Azure Data Explorer cluster. For aggregation options, see IngestionBatching policy.

If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. All community members with similar issues will benefit by doing so. Your contribution is highly appreciated.
Please sign in to rate this answer.
Javier Huamani 20 Reputation points

2024-02-23T15:38:25.62+00:00

Hi Sander, thanks for following up. I've set and enabled the streamingingestion policy for my ADX Table and the Cluster has been configured for streaming. However, ASA data is still coming in to the table at a default 5 minute intervals. Neither the table or database have ingestionbatching policies. I'm not sure why the streamingingestion policy is being ignored.

Sander van de Velde 28,386 Reputation points MVP

2024-02-23T23:12:28.2233333+00:00

Hello @Javier Huamani

Make sure you first enable streaming ingestion on the cluster:

Then, for each table ingesting data, enable streaming ingestion:

.alter table Telemetry policy streamingingestion enable

Use the right table names.

Notice this can take a couple of minutes before this is activated (and the underlying infrastructure is up and running).

Please restart the Stream Analytics job to make sure everything works as expected.

Does this help?

If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. All community members with similar issues will benefit by doing so. Your contribution is highly appreciated.

Javier Huamani 20 Reputation points

2024-02-26T15:55:00.6366667+00:00

I followed your steps and I restarted the Stream job multiple times. However, the ADX output still defaulted to batch ingestion of 5 minutes. I found an alternate solution by outputting the same data to an Event Hub and then connecting a new Staging table in ADX to the EH. I enabled the streamingingestion for the new table and now the data is streaming without issues.

Javier Huamani 20 Reputation points

2024-02-26T16:08:09.4166667+00:00

I'm going to use the alternative solution with an EH for now since I need the streaming to work.

Sander van de Velde 28,386 Reputation points MVP

2024-02-26T18:20:41.33+00:00

Hello @Javier Huamani

Yes, using the eventhub in between is working, I use is for multiple projects.

The Event Hub offers better insights on what is outputted by Stream Analytics and you can pause the eventhub if needed for ADX maintenance.

It also gives you more control on the table mapping.

The answer provided above should lead to the right outcome though, as documented.

If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. All community members with similar issues will benefit by doing so. Your contribution is highly appreciated.
Sign in to comment

How to achieve streaming ingestion for ASA input to ADX?

0 additional answers