Azure Iothub, Total devices growing without us actually adding a device

Collin Koornstra 11

For a IoT project we use IoTHub to connect our devices to the cloud. The last week we noticed that our device count randomly grew everynight. Its seems like a random device is connected and after 10 minutes or less it automaticly disconnects.
We're currently testing a use-case to see if IoTHub could be used in our production environment but before we make the choice we would like to know why there are random devices appearing.

As you can see in this image our total devices started spiking everynight since this weekend.

Is anyone else experiencing the same issue?

AshokPeddakotla-MSFT 27,311 Reputation points

2020-10-07T14:34:06.46+00:00

@Collin Koornstra Welcome to Q&A forum!

Sorry to hear that you are having trouble with the spikes.

Could you please share more details on how you are connecting the devices with IoT Hub and what code are you using in the simulated devices?
Also, kindly check Monitor the health of Azure IoT Hub and see if it helps to identify the root cause of spike.

You can also Control access to IoT Hub
Sander van de Velde 28,311 Reputation points MVP

2020-10-09T07:56:51.527+00:00

I checked my own IoT Hub and I see a similar pattern for the last 24 hours (where I did not add or delete any device):

This metric is of course still in preview (as seen in the image), but it's a bit strange it's not a flat line.
AshokPeddakotla-MSFT 27,311 Reputation points

2020-10-09T12:41:43.68+00:00

@Sander van de Velde Sorry for the inconvenience caused.

I have reached out to our internal team on the issue and once I have an update will keep you posted on same.
Collin Koornstra 11 Reputation points

2020-10-12T08:24:41.463+00:00

@AshokPeddakotla-MSFT Thanks for the answer, I've checked the health of the IoT Hub but nothing special happens.
I will wait for your update towards @Sander van de Velde
Sander van de Velde 28,311 Reputation points MVP

2020-10-12T18:16:13.74+00:00

I just checked the metrics again:

There is something strange going on. This is an IoTHub which is running for a long time already in West Europe.

I have not added or deleted any device today.

Note: 'Total devices' is not in preview anymore.
Collin Koornstra 11 Reputation points

2020-10-13T06:32:45.23+00:00

@Sander van de Velde This is the same issue we experience, but after every drop we also see a small increase.
António Sérgio Azevedo 7,661 Reputation points Microsoft Employee

2020-10-13T10:23:37.277+00:00

Hello,
I believe this could be a similar case where we confirmed that the API is eventually consistent which could cause the abnormal Total Devices showing in the dashboard: https://github.com/MicrosoftDocs/azure-docs/issues/40863

Though please bear with me while we confirm with Product Group if this is expected since it is currently in preview.
António Sérgio Azevedo 7,661 Reputation points Microsoft Employee

2020-10-13T10:32:13.577+00:00

I too see spikes when I select (Aggregation: Average) or (Aggregation: Min) - didn't add or delete devices as well recently:

(Aggregation: Max) shows the correct values:
António Sérgio Azevedo 7,661 Reputation points Microsoft Employee

2020-10-20T10:53:16.31+00:00

Quick update that we are internally investigating this issue with Product Group. Thanks for your time so far.

1 answer

António Sérgio Azevedo 7,661 Reputation points Microsoft Employee

2020-11-13T16:26:09.567+00:00
Update March 2021

=============

THIS IS NOW FIXED

You can now start using Total Devices Count Metric again.

PS: Ignore the average Aggregation in Total Devices Count metric and use Max only.

"Average of total device count, is aggregated slightly different from Max/Min. Max/min is taken whenever there is a snapshot data point, i.e. total device count in minutely interval; whereas average is computed upon sum/count. Depending on how many data points emitted per minute interval, there will be a small variance spread across the central line. Upon observing the wavy spike below, it is completely normal and should be tolerated."

Update November 2020

=============

Hello,
We have confirmed this is a bug and is currently being fixed. Please check the Root Cause Analysis and mark this as answer or let us know any concerns you may have while this is being fixed:

Root cause: We have identified a bug in service emitting a metric for total device count twice within a minute, resulting higher number of devices (than customer actually has) to shown in Total Device Count tile of the IoT Hub dashboard.
Mitigation: As a workaround, please use 'IoT Hub Usage' tile instead of 'Total Device Count'
Next Steps: We are working on fixing the concurrency issue with the tasks that update the 'totalDeviceCount' metric.

Thank you so much for your time @Collin Koornstra @Sander van de Velde

Remember:

Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.

Want a reminder to come back and check responses? Here is how to subscribe to a notification.
Please sign in to rate this answer.

1 person found this answer helpful.
António Sérgio Azevedo 7,661 Reputation points Microsoft Employee

2021-02-23T08:44:21.057+00:00

@Collin Koornstra @Sander van de Velde

Please can you validate (and mark as answer) that the issue is now resolved?

Thank you so much!

Remember:

Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.

Sander van de Velde 28,311 Reputation points MVP

2021-02-23T14:18:52.297+00:00

Hello,

The max number of devices does not have those spikes anymore (I only added an extra device a few minutes ago):

Still, the average is a bit strange, I expected a flat line (because I did not change the number of devices during the last 24 hours):

António Sérgio Azevedo 7,661 Reputation points Microsoft Employee

2021-03-11T02:17:33.23+00:00

@Sander van de Velde thanks for checking it. Looping back with Product Team. @Collin Koornstra can you share your results as well? Tanks!

António Sérgio Azevedo 7,661 Reputation points Microsoft Employee

2021-03-15T19:58:27.723+00:00

@Sander van de Velde @Collin Koornstra I have confirmed with product group that you should ignore the average Aggregation in Total Devices Count metric and use Max only.

"Average of total device count, is aggregated slightly different from Max/Min. Max/min is taken whenever there is a snapshot data point, i.e. total device count in minutely interval; whereas average is computed upon sum/count. Depending on how many data points emitted per minute interval, there will be a small variance spread across the central line. Upon observing the wavy spike below, it is completely normal and should be tolerated."

Please let me know if you are ok with this and mark the answer as verified.

Regards!
Sign in to comment