how to maintain multiple streams in single IOT Hub?

kumaraswamy gosukonda 1 Reputation point
2020-01-24T17:21:32.037+00:00

i have a situation like, we have single IOT hub which will receive compressed data and uncompressed data. we have single stream analytics service to process IOT hub messages, i have added two inputs in Azure Stream Analytics to process compressed and uncompressed data.

i just wanted to route compressed messages to one input (Stream Analytics) and uncompressed messages to another input to process both stream of data with single instance of IOT Hub and Stream Analytics.

Microsoft Entra ID
Microsoft Entra ID
A Microsoft Entra identity service that provides identity management and access control capabilities. Replaces Azure Active Directory.
19,640 questions
0 comments

2 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Marilee Turscak-MSFT 34,066 Reputation points Microsoft Employee
    2020-01-30T19:16:51.93+00:00

    Hi,

    Unfortunately this service is not supported in Q&A yet (slated to be added in May). You will need to ask on MSDN for now. https://social.msdn.microsoft.com/Forums/en-US/home?category=InternetOfThings

    I have alerted that team about this thread but they do not appear to have Q&A profiles set up.

    This thread looks somewhat related to your ask, though. https://stackoverflow.com/questions/58724735/stream-analytics-process-compressed-and-uncompressed-data-in-same-job

    0 comments
  2. Sander van de Velde 29,196 Reputation points MVP
    2020-04-20T15:58:43.083+00:00

    You need to define two separate inputs in your Stream Analytics job.

    Stream Analytics inputs you have to use, are either of type IoT Hub or Event Hub. Both support Event compression, like GZip or Deflate.

    If both the two types of messages were not compressed, you could connect the default, built-in endpoint to multiple inputs, each with their own consumer group and filter within the job.

    Due to the mixed compressed/not compressed telemetry situation, you have to split your IotHub telemetry before handing it over to the stream analytics job.

    This can be done using the IoTHub routing, together with an Eventhub:

    1. Create an eventhub namespace and one eventhub
    2. Create an iot hub custom endpoint and connect it to this eventhub
    3. Disable the fallback route in the IotHub (just to prevent mixing non-filtered messages with the compressed ones)
    4. Create a route with the default built-in endpoint, listening to compressed messages **)
    5. Create a route with the evenhub listening to non-compressed messages **)
    6. In the ASA job create an iothub input for the compressed messages
    7. In the ASA job create a second Eventhub input for the uncompressed messages

    **) You have to use the routing query to split the messages. I recommend using an application property like compressed='gzip' to split the incoming messages over the two routes.

    The Message class has a property bag for that. Splitting using a message body value is hard (not possible?) due to the compressed nature of messages (I expect, only the body is compressed).

    You could also try to accomplish this with two event hubs instead of one iothub endpoint and one eventhub.

    See also this tutorial.

    do not forget to mark the right answer

    (do not forget to mark the right answer)

    0 comments