Upload files from your device to the cloud with IoT Hub (Python)

This article shows how to use the file upload capabilities of IoT Hub to upload a file to Azure blob storage. The tutorial shows you how to:

  • Securely provide a storage container for uploading a file.

  • Use the Python client to upload a file through your IoT hub.

The Send telemetry from a device to an IoT hub quickstart demonstrates the basic device-to-cloud messaging functionality of IoT Hub. However, in some scenarios you cannot easily map the data your devices send into the relatively small device-to-cloud messages that IoT Hub accepts. When you need to upland files from a device, you can still use the security and reliability of IoT Hub.

At the end of this tutorial, you run the Python console app:

  • FileUpload.py, which uploads a file to storage using the Python Device SDK.

Note

IoT Hub has SDK support for many device platforms and languages (including C, Java, Javascript, and Python) through Azure IoT device SDKs. For instructions on how to use Python to connect your device to this tutorial's code, and generally to Azure IoT Hub, see the Azure IoT Python SDK.

Important

File upload functionality on devices that use X.509 certificate authority (CA) authentication is in public preview, and preview mode must be enabled. It is generally available on devices that use X.509 thumbprint authentication. To learn more about X.509 authentication with IoT Hub, see Supported X.509 certificates.

Prerequisites

  • Make sure that port 8883 is open in your firewall. The device sample in this article uses MQTT protocol, which communicates over port 8883. This port may be blocked in some corporate and educational network environments. For more information and ways to work around this issue, see Connecting to IoT Hub (MQTT).

Create an IoT hub

This section describes how to create an IoT hub using the Azure portal.

  1. Sign in to the Azure portal.

  2. From the Azure homepage, select the + Create a resource button, and then enter IoT Hub in the Search the Marketplace field.

  3. Select IoT Hub from the search results, and then select Create.

  4. On the Basics tab, complete the fields as follows:

    • Subscription: Select the subscription to use for your hub.

    • Resource Group: Select a resource group or create a new one. To create a new one, select Create new and fill in the name you want to use. To use an existing resource group, select that resource group. For more information, see Manage Azure Resource Manager resource groups.

    • Region: Select the region in which you want your hub to be located. Select the location closest to you. Some features, such as IoT Hub device streams, are only available in specific regions. For these limited features, you must select one of the supported regions.

    • IoT Hub Name: Enter a name for your hub. This name must be globally unique, with a length between 3 and 50 alphanumeric characters. The name can also include the dash ('-') character.

    Important

    Because the IoT hub will be publicly discoverable as a DNS endpoint, be sure to avoid entering any sensitive or personally identifiable information when you name it.

    Create a hub in the Azure portal.

  5. Select Next: Networking to continue creating your hub.

    Choose the endpoints that devices can use to connect to your IoT Hub. You can select the default setting Public endpoint (all networks), or choose Public endpoint (selected IP ranges), or Private endpoint. Accept the default setting for this example.

    Choose the endpoints that can connect.

  6. Select Next: Management to continue creating your hub.

    Set the size and scale for a new hub using the Azure portal.

    You can accept the default settings here. If desired, you can modify any of the following fields:

    • Pricing and scale tier: Your selected tier. You can choose from several tiers, depending on how many features you want and how many messages you send through your solution per day. The free tier is intended for testing and evaluation. It allows 500 devices to be connected to the hub and up to 8,000 messages per day. Each Azure subscription can create one IoT hub in the free tier.

      If you are working through a Quickstart for IoT Hub device streams, select the free tier.

    • IoT Hub units: The number of messages allowed per unit per day depends on your hub's pricing tier. For example, if you want the hub to support ingress of 700,000 messages, you choose two S1 tier units. For details about the other tier options, see Choosing the right IoT Hub tier.

    • Defender for IoT: Turn this on to add an extra layer of threat protection to IoT and your devices. This option is not available for hubs in the free tier. For more information about this feature, see Azure Defender for IoT.

    • Advanced Settings > Device-to-cloud partitions: This property relates the device-to-cloud messages to the number of simultaneous readers of the messages. Most hubs need only four partitions.

  7. Select Next: Tags to continue to the next screen.

    Tags are name/value pairs. You can assign the same tag to multiple resources and resource groups to categorize resources and consolidate billing. In this document, you won't be adding any tags. For more information, see Use tags to organize your Azure resources.

    Assign tags for the hub using the Azure portal.

  8. Select Next: Review + create to review your choices. You see something similar to this screen, but with the values you selected when creating the hub.

    Review information for creating the new hub.

  9. Select Create to start the deployment of your new hub. Your deployment will be in progress a few minutes while the hub is being created. Once the deployment is complete, click Go to resource to open the new hub.

Register a new device in the IoT hub

In this section, you create a device identity in the identity registry in your IoT hub. A device cannot connect to a hub unless it has an entry in the identity registry. For more information, see the IoT Hub developer guide.

  1. In your IoT hub navigation menu, open IoT Devices, then select New to add a device in your IoT hub.

    Create device identity in portal

  2. In Create a device, provide a name for your new device, such as myDeviceId, and select Save. This action creates a device identity for your IoT hub. Leave Auto-generate keys checked so that the primary and secondary keys will be generated automatically.

    Add a new device

    Important

    The device ID may be visible in the logs collected for customer support and troubleshooting, so make sure to avoid any sensitive information while naming it.

  3. After the device is created, open the device from the list in the IoT devices pane. Copy the Primary Connection String. This connection string is used by device code to communicate with the hub.

    By default, the keys and connection strings are masked as they are sensitive information. If you click the eye icon, they are revealed as shown in the image below. It is not necessary to reveal them to copy them with the copy button.

    Device connection string

Note

The IoT Hub identity registry only stores device identities to enable secure access to the IoT hub. It stores device IDs and keys to use as security credentials, and an enabled/disabled flag that you can use to disable access for an individual device. If your application needs to store other device-specific metadata, it should use an application-specific store. For more information, see IoT Hub developer guide.

Associate an Azure Storage account to IoT Hub

To upload files from a device, you must have an Azure Storage account and Azure Blob Storage container associated with your IoT hub. Once you associate the storage account and container with your IoT hub, your IoT hub can provide the elements of a SAS URI when requested by a device. The device can then use these elements to construct the SAS URI that it uses to authenticate with Azure Storage and upload files to the blob container.

To associate an Azure Storage account with your IoT hub:

  1. Under Messaging, select File Upload on the left-pane of your IoT hub.

    Select file upload settings from the portal.

  2. On the File upload pane, select Azure Storage Container. For this article, it's recommended that your storage account and IoT Hub be located in the same region.

    • If you already have a storage account you want to use, select it from the list.

    • To create a new storage account, select +Storage account. Provide a name for the storage account and make sure the Location is set to the same region as your IoT hub, then select OK. The new account is created in the same resource group as your IoT hub. When the deployment completes, select the storage account from the list.

    After you select the storage account, the Containers pane opens.

  3. On the Containers pane, select the blob container.

    • If you already have a blob container you want to use, select it from the list and click Select.

    • To create a new blob container, select + Container. Provide a name for the new container. For the purposes of this article, you can leave all other fields at their default. Select Create. When the deployment completes, select the container from the list and click Select.

  4. Back on the File upload pane, make sure that file notifications are set to On. You can leave all other settings at their defaults. Select Save and wait for the settings to complete before moving on to the next section.

    Confirm file upload settings in the portal.

For more detailed instructions on how to create an Azure Storage account, see Create a storage account. For more detailed instructions on how to associate a storage account and blob container with an IoT hub, see Configure file uploads using the Azure portal.

Upload a file from a device app

In this section, you create the device app to upload a file to IoT hub.

  1. At your command prompt, run the following command to install the azure-iot-device package. You use this package to coordinate the file upload with your IoT hub.

    pip install azure-iot-device
    
  2. At your command prompt, run the following command to install the azure.storage.blob package. You use this package to perform the file upload.

    pip install azure.storage.blob
    
  3. Create a test file that you'll upload to blob storage.

  4. Using a text editor, create a FileUpload.py file in your working folder.

  5. Add the following import statements and variables at the start of the FileUpload.py file.

    import os
    from azure.iot.device import IoTHubDeviceClient
    from azure.core.exceptions import AzureError
    from azure.storage.blob import BlobClient
    
    CONNECTION_STRING = "[Device Connection String]"
    PATH_TO_FILE = r"[Full path to local file]"
    
  6. In your file, replace [Device Connection String] with the connection string of your IoT hub device. Replace [Full path to local file] with the path to the test file that you created or any file on your device that you want to upload.

  7. Create a function to upload the file to blob storage:

    def store_blob(blob_info, file_name):
        try:
            sas_url = "https://{}/{}/{}{}".format(
                blob_info["hostName"],
                blob_info["containerName"],
                blob_info["blobName"],
                blob_info["sasToken"]
            )
    
            print("\nUploading file: {} to Azure Storage as blob: {} in container {}\n".format(file_name, blob_info["blobName"], blob_info["containerName"]))
    
            # Upload the specified file
            with BlobClient.from_blob_url(sas_url) as blob_client:
                with open(file_name, "rb") as f:
                    result = blob_client.upload_blob(f, overwrite=True)
                    return (True, result)
    
        except FileNotFoundError as ex:
            # catch file not found and add an HTTP status code to return in notification to IoT Hub
            ex.status_code = 404
            return (False, ex)
    
        except AzureError as ex:
            # catch Azure errors that might result from the upload operation
            return (False, ex)
    

    This function parses the blob_info structure passed into it to create a URL that it uses to initialize an azure.storage.blob.BlobClient. Then it uploads your file to Azure blob storage using this client.

  8. Add the following code to connect the client and upload the file:

    def run_sample(device_client):
        # Connect the client
        device_client.connect()
    
        # Get the storage info for the blob
        blob_name = os.path.basename(PATH_TO_FILE)
        storage_info = device_client.get_storage_info_for_blob(blob_name)
    
        # Upload to blob
        success, result = store_blob(storage_info, PATH_TO_FILE)
    
        if success == True:
            print("Upload succeeded. Result is: \n") 
            print(result)
            print()
    
            device_client.notify_blob_upload_status(
                storage_info["correlationId"], True, 200, "OK: {}".format(PATH_TO_FILE)
            )
    
        else :
            # If the upload was not successful, the result is the exception object
            print("Upload failed. Exception is: \n") 
            print(result)
            print()
    
            device_client.notify_blob_upload_status(
                storage_info["correlationId"], False, result.status_code, str(result)
            )
    
    def main():
        device_client = IoTHubDeviceClient.create_from_connection_string(CONNECTION_STRING)
    
        try:
            print ("IoT Hub file upload sample, press Ctrl-C to exit")
            run_sample(device_client)
        except KeyboardInterrupt:
            print ("IoTHubDeviceClient sample stopped")
        finally:
            # Graceful exit
            device_client.shutdown()
    
    
    if __name__ == "__main__":
        main()
    

    This code creates an IoTHubDeviceClient and uses the following APIs to manage the file upload with your IoT hub:

    • get_storage_info_for_blob gets information from your IoT hub about the linked Storage Account you created previously. This information includes the hostname, container name, blob name, and a SAS token. The storage info is passed to the store_blob function (created in the previous step), so the BlobClient in that function can authenticate with Azure storage. The get_storage_info_for_blob method also returns a correlation_id, which is used in the notify_blob_upload_status method. The correlation_id is IoT Hub's way of marking which blob you're working on.

    • notify_blob_upload_status notifies IoT Hub of the status of your blob storage operation. You pass it the correlation_id obtained by the get_storage_info_for_blob method. It's used by IoT Hub to notify any service that might be listening for a notification on the status of the file upload task.

  9. Save and close the FileUpload.py file.

Run the application

Now you're ready to run the application.

  1. At a command prompt in your working folder, run the following command:

    python FileUpload.py
    
  2. The following screenshot shows the output from the FileUpload app:

    Output from simulated-device app

  3. You can use the portal to view the uploaded file in the storage container you configured:

    Uploaded file

Next steps

In this tutorial, you learned how to use the file upload capabilities of IoT Hub to simplify file uploads from devices. You can continue to explore IoT hub features and scenarios with the following articles:

Learn more about Azure Blob Storage with the following links: