How to fix this error "The streaming job failed: Stream Analytics job is only able to use up to 50 Streaming Units based on the provided query".Please adjust the number of Streaming Units or check the number of input partitions and update your query.

Question

I have been experiencing this when I try to run Azure Stream Analytics v2 with 6 SUs or above .Below 6 SU it works fine. I have a simple query which only has CROSS APPLY and GET ARRAY ELEMENTS in it.I am fetching telemetry data arrays and converting them in rows and storing in SQL DB.

Answer 1

@Rafay Khan - Thanks for the question and using MS Q&A platform.

The error message you are seeing indicates that your Azure Stream Analytics job is only able to use up to 50 Streaming Units based on the provided query. This means that your query is not optimized for scaling beyond 50 Streaming Units.

To resolve this issue, you can consider the following options:

1. Optimize your query: Review your query and optimize it for scaling beyond 50 Streaming Units. You can use the Azure Stream Analytics Query Performance Optimization guide to help you optimize your query.
2. Increase the number of input partitions: If your input source has fewer partitions than the number of Streaming Units you are trying to use, you can increase the number of input partitions to match the number of Streaming Units. This can help improve the scalability of your job.
3. Increase the number of Streaming Units gradually: If you are experiencing issues when using 6 or more Streaming Units, you can try increasing the number of Streaming Units gradually. For example, you can start with 6 Streaming Units and gradually increase to 12, 18, 24 and so on until you reach the desired number of Streaming Units.
4. Consider using a different output sink: If you are storing data in SQL DB, you can consider using a different output sink that is optimized for high throughput and scalability, such as Azure Cosmos DB or Azure Data Lake Storage.

By following these options, you can resolve the issue and scale your Azure Stream Analytics job beyond 50 Streaming Units.

Streaming Units (SUs) represents the computing resources that are allocated to execute a Stream Analytics job. The higher the number of SUs, the more CPU and memory resources are allocated for your job. This capacity lets you focus on the query logic and abstracts the need to manage the hardware to run your Stream Analytics job in a timely manner.

Azure Stream Analytics supports two streaming unit structures: SU V1(to be deprecated) and SU V2(recommended).

The SU V1 model is ASA's original offering where every 6 SUs correspond to a single streaming node for a job. Jobs may run with 1 and 3 SUs as well and these correspond with fractional streaming nodes. Scaling occurs in increments of 6 beyond 6 SU jobs, to 12, 18, 24 and beyond by adding more streaming nodes that provide distributed computing resources.

Note: You need to add PARTITION BY clause to be able to use more than 6 SUs.

For more details, refer to Understand and adjust Stream Analytics streaming units and Scale an Azure Stream Analytics job to increase throughput.

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.