question

AleksanderV avatar image
0 Votes"
AleksanderV asked asergaz edited

IoT Hub reliability

Hi,

I'm having reliability issues with the IoT Hub. I wrote a very simple application that sends ten messages per second to an S2 Hub (which should be able to handle 70 messages per second). The timeout on messages is set to 2 seconds. Sending 500 messages, and getting about 44 errors: {'InvalidStateError': 37, 'ConnectionDroppedError': 3, 'Connection reset by peer': 4, 'Timeout': 37}. Is there something wrong with the code, or is this expected behavior based on the current level of service?

 import asyncio
 import sys
    
 from azure.iot.device.aio import IoTHubDeviceClient
    
    
 # IoTHubDeviceClient reports errors to stderr. Using this class to grab and count them before printing
 class Redirect_stderr:
     stderr = sys.stderr
     errors_to_grab = {
         "InvalidStateError": 0,
         "ConnectionDroppedError": 0,
         "Connection reset by peer": 0,
         "Timeout": 0,
     }
     collect_errors = ""
    
     def __enter__(self):
         sys.stderr = self
    
     def __exit__(self, exc_type, exc_val, exc_tb):
         sys.stderr = self.stderr
         print(self.collect_errors, file=sys.stderr)
         sys.stderr.flush()
         print(self.errors_to_grab)
    
     def write(self, text):
         for error in self.errors_to_grab.keys():
             if error in text:
                 self.errors_to_grab[error] += 1
         self.collect_errors += text
    
    
 MIN_DELAY_BETWEEN_MESSAGES = 0.1
 MESSAGE_TIMEOUT = 2
 N_MESSAGES = 500
    
 async def main():
     with Redirect_stderr():
         conn_str = "HostName=IoTHubForTest23.azure-devices.net;DeviceId=testDevice;SharedAccessKey=***redacted***"
         client = IoTHubDeviceClient.create_from_connection_string(conn_str)
         await client.connect()
         for i in range(N_MESSAGES):
             try:
                 print(f"message count: {i}")
                 task = asyncio.create_task(client.send_message(f"Message {i}"))
                 await asyncio.sleep(MIN_DELAY_BETWEEN_MESSAGES) # IoT Hub B1 has a message limit of four per second
                 await asyncio.wait_for(task, MESSAGE_TIMEOUT - MIN_DELAY_BETWEEN_MESSAGES)
             except asyncio.TimeoutError:
                 print("Timeout", file=sys.stderr)
    
    
 if __name__ == "__main__":
     asyncio.run(main())


azure-iot-hub
· 4
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @AleksanderV ,
I would like to call the attention that an S2 IoTHub can handle 120 Device-to-cloud send operations/sec/unit and not 70 like you mention.
See: https://docs.microsoft.com/en-us/azure/iot-hub/iot-hub-devguide-quotas-throttling

Could you be more specific on the environment you are running the code sample shared above? Is it on an Azure VM? What is the Azure IoT SDK Version?

Thank you!



0 Votes 0 ·

Unfortunately, this doesn't look any better on an IPC. {'InvalidStateError': 6, 'ConnectionDroppedError': 0, 'Connection reset by peer': 0, 'Timeout': 25}

1 Vote 1 ·

Hello @asergaz !

Thanks for following up on this, the platform is otherwise stable, so I hope this is a fluke of some sort. I am running on laptop computer with Ubuntu, not unlike the Industrial PC on which we intend to deploy. Further,

 $ head -n2 /etc/os-release 
 NAME="Ubuntu"
 VERSION="20.04.2 LTS (Focal Fossa)"

 $ python --version
 Python 3.9.0+

 $ pip freeze
 appdirs==1.4.4
 azure-iot-device==2.5.1
 azure-iot-hub==2.2.3
 black==21.4b2
 certifi==2020.12.5
 chardet==4.0.0
 click==7.1.2
 deprecation==2.1.0
 idna==2.10
 isodate==0.6.0
 isort==5.8.0
 janus==0.4.0
 msrest==0.6.21
 mypy-extensions==0.4.3
 oauthlib==3.1.0
 packaging==20.9
 paho-mqtt==1.5.1
 pathspec==0.8.1
 pyparsing==2.4.7
 PySocks==1.7.1
 regex==2021.4.4
 requests==2.25.1
 requests-oauthlib==1.3.0
 requests-unixsocket==0.2.0
 six==1.15.0
 toml==0.10.2
 uamqp==1.3.0
 urllib3==1.25.11
0 Votes 0 ·

The internet connectivity is stable. Packet losses do happen, but very rarely. Here is the output from flood pinging AT&T (from Europe):

 $ sudo ping -f att.com -c1000 -s1000
 PING att.com (144.160.36.42) 1000(1028) bytes of data.
              
 --- att.com ping statistics ---
 1000 packets transmitted, 1000 received, 0% packet loss, time 15227ms
 rtt min/avg/max/mdev = 113.407/120.483/145.926/1.786 ms, pipe 10, ipg/ewma 15.242/120.642 ms


0 Votes 0 ·

1 Answer

asergaz avatar image
0 Votes"
asergaz answered asergaz edited

@AleksanderV thank you for sharing more details.

We may need to look at server side logs to understand better how IoTHub is responding to your D2C messages. I don't immediately see anything wrong with the piece of code you shared. Please Create an Azure support request, reference this issue and let me know the number of the ticket.

Let me also suggest that you look at this related thread: IoT Hub does not allow near real time messaging

"For IoT Hub, we promise that at least 99.9% of the time deployed IoT hubs will be able to send messages to and receive messages from registered devices and the Service will able to perform create, read, update, and delete operations on IoT hubs."

In a 30day billing period the SLA is met even if we have a downtime\delay of around 43 minutes. IoTHub didn't achieve 100% Service Level Agreemen (SLA) yet and messages lost can happen (you will need to prepare your application for that): SLA for Azure IoT Hub

Thanks!

Remember:
- Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.
- Want a reminder to come back and check responses? Here is how to subscribe to a notification.


5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.