question

VijayKarthikG-5711 avatar image
0 Votes"
VijayKarthikG-5711 asked SandervandeVelde42 edited

ParitionKey is null in EventHub Messages when routed from IotHub

We have set up IoT architecture where we have multiple device sending MQTT messages to Iot Hub. Then the telemetry messages from IoT Hub is routed to a event hub(custom) endpoint. The event hub has four partitions .
We are consuming the messages using default consumer group. The question I have is how the device telemetry will be stored across the partitions ?.If it is based on a hash function on DeviceId , the partition key will be unique for 1000's of devices , so in that case whether a single partition is sufficient or your suggestion would be using multiple partitions(i.e Is there any chance of performance improvement if we use multiple partitions).One more question is we are reading event from Event Hub using Event Processor client , the retrieved events have the PartitionKey Values as null. Is there any specific reason for this null value in retrived events?

azure-iot-hubazure-event-hubs
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@VijayKarthikG-5711 Wanted to follow up with you to understand whether the below information helped. Please let us know if there are any other queries.

If an answer is helpful, please click on 130616-image.png or upvote 130671-image.png which might help other community members reading this thread.

0 Votes 0 ·

1 Answer

SandervandeVelde42 avatar image
0 Votes"
SandervandeVelde42 answered SandervandeVelde42 edited

Hello @VijayKarthikG-5711 ,

partitionkeys are used to streamline and scale resources like the iothub and eventhub. Once you have chosen a certain number of partitions (4 by default for an IoT Hub) it's normal to use that same number downstream (like in the evenhub). Otherwise, it acts as a funnel, blocking ingestion.

Normally, 4 partitions should be enough for the majority of customers. See this documentation with background information about choosing the number of partitions.

Regarding the null values, I did a quick experiment.

I created an IoT Hub with 4 partitions:

145866-image.png

I also created an eventhub with 4 partitions:

145857-image.png

A custom endpoint and route is added to send data to the eventhub:

145867-image.png

I send messages from a simulated device:

145865-image.png

The Eventhub offers a quick insight into the incoming data using the Process data menu:

145883-image.png

Explore:

145891-image.png

Here you see the partition key in the query editor:

145816-image.png

So technically, the partitioning seems to work out well. Please double-check your code.



image.png (6.4 KiB)
image.png (12.4 KiB)
image.png (21.9 KiB)
image.png (29.7 KiB)
image.png (1.2 KiB)
image.png (13.8 KiB)
image.png (27.4 KiB)
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @SandervandeVelde42 sorry but I do not understand: at the env you wrote "Here you see the partition key in the query editor:" and you paste the image of ASA input....but i cannot see the "partition key".
I also try to reproduce your example but Partition Key is still null/empty and I do not find a way to force it (also from code)
Am I missing something?

0 Votes 0 ·
SandervandeVelde42 avatar image SandervandeVelde42 SimoneCherici-5046 ·

Hello @SimoneCherici-5046 ,

well, the PartitionKey is actually that PartitionId (second column from the right).

Here is the raw view from another simulator device:

165822-image.png

You need to take care about the number of partitions!

Notice that the Azure Portal does not provide the best debugging behavior. For example, system properties are not made available in that dialog.

Check out this blog post to see how you can use an Azure Function to have full control over (non-) telemetry messages sent to the Event Hub or getting them directly from the IoT Hub.



0 Votes 0 ·
image.png (62.8 KiB)