Azure Stream Analytics Job has water mark delay consistenly over 1 hour

Anand Pangare (Tata Consultancy Services) 1 Microsoft Vendor

Hi,

We are consistently facing a watermark delay issue in one of our stream analytics job in productions, the SU% utilization seems below 10%. Due to this there is a latency of up to 4 hours in our NRT pipeline. The Stream analytic job reads from an Event Hub with 100 partitions and drop events to an ADLS Cosmos end point. The query is partitioned too by partition id.

SELECT
    *
INTO
    ADLS
FROM
    EH
    PARTITION BY PartitionId

The stream analytics job receives almost 9.7TB data every day. The stream analytics job has a very high backlog of input events. The streaming units is also set on a higher side to 186 SU. Could you please let us know how we can resolve the issue?

HimanshuSinha-msft 19,381 Reputation points Microsoft Employee

2020-10-15T22:39:18.957+00:00

Hello @Anand Pangare (Tata Consultancy Services) ,

Thanks for the ask and also using the Microsoft Q&A.

Not sure if you are leveraging the query paralleization as called out
https://learn.microsoft.com/en-us/azure/stream-analytics/stream-analytics-parallelization#embarrassingly-parallel-jobs.
Let me know how it goes .

Thanks Himanshu
Anand Pangare (Tata Consultancy Services) 1 Reputation point Microsoft Vendor

2020-10-15T22:44:03.01+00:00

Hi @HimanshuSinha-msft ,

Thanks for responding, could you please elaborate more on what do you mean by leveraging parallelization. I am using EH
PARTITION BY PartitionId in my query, would this not suffice as a simple parallelization.

Looking forward to your input.

Thanks,
Anand
HimanshuSinha-msft 19,381 Reputation points Microsoft Employee

2020-10-19T20:53:48.457+00:00

Hello @Anand Pangare (Tata Consultancy Services) ,

Thanks for the reply .

Just adding the patitionid will not do the trick. Please do read the point#2
https://learn.microsoft.com/en-us/azure/stream-analytics/stream-analytics-parallelization#embarrassingly-parallel-jobs. Also the #3 talks
about the partition on the output side . I am not sure ADLS cosmos helps with that .

Thanks Himanshu

1 answer

JS Azure 76 Reputation points

2020-10-20T22:56:22.343+00:00

Hi,
I see that you have "partition by" in the input and you use ADLS as output (which supports partitioning), so you are on the right track to scale out.
What can be confusing with the SU% is that it only reflects the memory used. So if the job is CPU-bound, you may not see it from this metric alone. A good indication of CPU limitation is when you see the number of backlogged events growing.
In this case I will recommend raising the numbers of SUs to see if the job is catching up.
Let me know if it works, happy to investigate this further.
Thanks,
JS
Please sign in to rate this answer.
Anand Pangare (Tata Consultancy Services) 1 Reputation point Microsoft Vendor

2020-10-21T17:54:02.703+00:00

Hi @JS Azure ,

Thanks for your response. Yes, I do see that the input events backlog grow as the water mark delay increases. I already have the job running at 186 SU which I think is already pretty high, I am looking for a solution where the pipeline would be more resilient and would be able to handle the data load in future.

Looking forward for your input.

Thanks,
Anand

JS Azure 76 Reputation points

2020-10-21T17:59:41.903+00:00

Sorry, just noticed the number of SU you use.
I actually see your support ticket, let's continue the investigation there. Our team can look if there is any bottleneck.

Anand Pangare (Tata Consultancy Services) 1 Reputation point Microsoft Vendor

2020-10-21T18:08:13.85+00:00

Thanks @JS Azure .
Sign in to comment

Share via

Azure Stream Analytics Job has water mark delay consistenly over 1 hour

1 answer