Tutorial: Create a hierarchy of IoT Edge devices (Preview)

Deploy Azure IoT Edge nodes across networks organized in hierarchical layers. Each layer in a hierarchy is a gateway device that handles messages and requests from devices in the layer beneath it.

Note

This feature requires IoT Edge version 1.2, which is in public preview, running Linux containers.

You can structure a hierarchy of devices so that only the top layer has connectivity to the cloud, and the lower layers can only communicate with adjacent north and south layers. This network layering is the foundation of most industrial networks, which follow the ISA-95 standard.

The goal of this tutorial is to create a hierarchy of IoT Edge devices that simulates a production environment. At the end, you will deploy the Simulated Temperature Sensor module to a lower layer device without internet access by downloading container images through the hierarchy.

To accomplish this goal, this tutorial walks you through creating a hierarchy of IoT Edge devices, deploying IoT Edge runtime containers to your devices, and configuring your devices locally. In this tutorial, you learn how to:

  • Create and define the relationships in a hierarchy of IoT Edge devices.
  • Configure the IoT Edge runtime on the devices in your hierarchy.
  • Install consistent certificates across your device hierarchy.
  • Add workloads to the devices in your hierarchy.
  • Use an API proxy module to securely route HTTP traffic over a single port from your lower layer devices.

In this tutorial, the following network layers are defined:

  • Top layer: IoT Edge devices at this layer can connect directly to the cloud.

  • Lower layer: IoT Edge devices at this layer cannot connect directly to the cloud. They need to go through one or more intermediary IoT Edge devices to send and receive data.

This tutorial uses a two device hierarchy for simplicity. One device, topLayerDevice, represents a device at the top layer of the hierarchy, which can connect directly to the cloud. This device will also be referred to as the parent device. The other device, lowerLayerDevice, represents a device at the lower layer of the hierarchy, which cannot connect directly to the cloud. This device will also be referred to as the child device. You can add additional lower layer devices to represent your production environment. The configuration of any additional lower layer devices will follow lowerLayerDevice's configuration.

Prerequisites

To create a hierarchy of IoT Edge devices, you will need:

  • A computer (Windows or Linux) with internet connectivity.

  • An Azure account with a valid subscription. If you don't have an Azure subscription, create a free account before you begin.

  • A free or standard tier IoT Hub in Azure.

  • Azure CLI v2.3.1 with the Azure IoT extension v0.10.6 or higher installed. This tutorial uses the Azure Cloud Shell. If you're unfamiliar with the Azure Cloud Shell, check out a quickstart for details.

  • Two Linux devices to configure as IoT Edge devices. If you don't have devices available, you can create two Azure virtual machines by replacing the placeholder text in the following command and running it twice:

    az vm create \
      --resource-group <REPLACE_WITH_RESOURCE_GROUP> \
      --name <REPLACE_WITH_UNIQUE_NAMES_FOR_EACH_VM> \
      --image UbuntuLTS \
      --admin-username azureuser \
      --admin-password <REPLACE_WITH_PASSWORD>
    
  • Make sure that the following ports are open inbound: 8000, 443, 5671, 8883:

    • 8000: Used to pull Docker container images through the API proxy.
    • 443: Used between parent and child edge hubs for REST API calls.
    • 5671, 8883: Used for AMQP and MQTT.

    For more information, see How to open ports to a virtual machine with the Azure portal.

You can also try out this scenario by following the scripted Azure IoT Edge for Industrial IoT sample, which deploys Azure virtual machines as preconfigured devices to simulate a factory environment.

Configure your IoT Edge device hierarchy

Create a hierarchy of IoT Edge devices

The first step, creating your IoT Edge devices, can be done through the Azure portal or Azure CLI. This tutorial will create a hierarchy of two IoT Edge devices: topLayerDevice and its child lowerLayerDevice.

  1. In the Azure portal, navigate to your IoT Hub.

  2. From the menu on the left pane, under Automatic Device Management, select IoT Edge.

  3. Select + Add an IoT Edge device. This device will be the top layer device, so enter an appropriate unique device ID. Select Save.

  4. Select + Add an IoT Edge device again. This device will be the edge lower layer device, so enter an appropriate unique device ID.

  5. Choose Set a parent device, choose your top layer device from the list of devices, and select Ok. Select Save.

    Setting parent for lower layer device

Make note of each IoT Edge device's connection string. They will be used later.

  1. In the Azure portal, navigate to the IoT Edge section of your IoT Hub.

  2. Click on the device ID of one of the devices in the list of devices.

  3. Select Copy on the Primary Connection String field and record it in a place of your choice.

  4. Repeat for all other devices.

Create certificates

All devices in a gateway scenario need a shared certificate to set up secure connections between them. Use the following steps to create demo certificates for both devices in this scenario.

To create demo certificates on a Linux device, you need to clone the generation scripts and set them up to run locally in bash.

  1. Clone the IoT Edge git repo, which contains scripts to generate demo certificates.

    git clone https://github.com/Azure/iotedge.git
    
  2. Navigate to the directory in which you want to work. All certificate and key files will be created in this directory.

  3. Copy the config and script files from the cloned IoT Edge repo into your working directory.

    cp <path>/iotedge/tools/CACertificates/*.cnf .
    cp <path>/iotedge/tools/CACertificates/certGen.sh .
    
  4. Create the root CA certificate and one intermediate certificate.

    ./certGen.sh create_root_and_intermediate
    

    This script command creates several certificate and key files, but we are using the following file as the root CA certificate for the gateway hierarchy:

    • <WRKDIR>/certs/azure-iot-test-only.root.ca.cert.pem
  5. Create two sets of IoT Edge device CA certificates and private keys with the following command: one set for the top layer device and one set for the lower layer device. Provide memorable names for the CA certificates to distinguish them from each other.

    ./certGen.sh create_edge_device_ca_certificate "top-layer-device"
    ./certGen.sh create_edge_device_ca_certificate "lower-layer-device"
    

    This script command creates several certificate and key files, but we are using the following certificate and key pair on each IoT Edge device and referenced in the config.yaml file:

    • <WRKDIR>/certs/iot-edge-device-<CA cert name>-full-chain.cert.pem
    • <WRKDIR>/private/iot-edge-device-<CA cert name>.key.pem

Each device needs a copy of the root CA certificate and a copy of its own device CA certificate and private key. You can use a USB drive or secure file copy to move the certificates to each device.

  1. After the certificates are transferred, install the root CA for each device.

    sudo cp <path>/azure-iot-test-only.root.ca.cert.pem /usr/local/share/ca-certificates/azure-iot-test-only.root.ca.cert.pem.crt
    sudo update-ca-certificates
    

    This command should output that one certificate has been added in /etc/ssl/certs.

Install IoT Edge on the devices

Install IoT Edge by following these steps on both devices.

  1. Install the repository configuration that matches your device operating system.

    • Ubuntu Server 18.04:

      curl https://packages.microsoft.com/config/ubuntu/18.04/multiarch/prod.list > ./microsoft-prod.list
      
    • Raspberry Pi OS Stretch:

      curl https://packages.microsoft.com/config/debian/stretch/multiarch/prod.list > ./microsoft-prod.list
      
  2. Copy the generated list to the sources.list.d directory.

    sudo cp ./microsoft-prod.list /etc/apt/sources.list.d/
    
  3. Install the Microsoft GPG public key.

    curl https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > microsoft.gpg
    sudo cp ./microsoft.gpg /etc/apt/trusted.gpg.d/
    
  4. Update package lists on your device.

    sudo apt-get update
    
  5. Install the Moby engine.

    sudo apt-get install moby-engine
    
  6. Install the hsmlib and IoT Edge daemon. To see the assets for other Linux distributions, visit the GitHub release.

    curl -L https://github.com/Azure/azure-iotedge/releases/download/1.2.0-rc1/libiothsm-std_1.2.0_rc1-1-1_debian9_amd64.deb -o libiothsm-std.deb
    curl -L https://github.com/Azure/azure-iotedge/releases/download/1.2.0-rc1/iotedge_1.2.0_rc1-1_debian9_amd64.deb -o iotedge.deb
    sudo dpkg -i ./libiothsm-std.deb
    sudo dpkg -i ./iotedge.deb
    

Configure the IoT Edge runtime

Configure the IoT Edge runtime by following these steps on both your devices. Configuring the IoT Edge runtime for your devices consists of four steps, all accomplished by editing the IoT Edge configuration file:

  1. Manually provision each device by adding that device's connection string to the configuration file.

  2. Finish setting up your device's certificates by pointing the configuration file to the device CA certificate, device CA private key, and root CA certificate.

  3. Bootstrap the system using the IoT Edge agent.

  4. Update the hostname parameter for your top layer device, and update both the hostname parameter and parent_hostname parameter for your lower layer devices.

Complete these steps and restart the IoT Edge service to configure your devices.

  1. On each device, open the IoT Edge configuration file.

    sudo nano /etc/iotedge/config.yaml
    
  2. Find the provisioning configurations of the file and uncomment the Manual provisioning configuration using a connection string section.

  3. Update the value of device_connection_string with the connection string from your IoT Edge device. Make sure any other provisioning sections are commented out. Make sure the provisioning: line has no preceding whitespace and that nested items are indented by two spaces.

    # Manual provisioning configuration using a connection string
    provisioning:
      source: "manual"
      device_connection_string: "<ADD DEVICE CONNECTION STRING HERE>"
      dynamic_reprovisioning: false
    

    Tip

    To paste clipboard contents into Nano Shift+Right Click or press Shift+Insert.

  4. Find the certificates section. Uncomment and update the three certificate fields to point to the certificates that you created in the previous section and moved to the IoT Edge device. Provide the file URI paths, which take the format file:///<path>/<filename>.

    certificates:
      device_ca_cert: "<File URI path to the device CA certificate unique to this device.>"
      device_ca_pk: "<File URI path to the device CA private key unique to this device.>"
      trusted_ca_certs: "<File URI path to the root CA certificate shared by all devices in the gateway hierarchy.>"
    
  5. For your top layer device, find the hostname parameter. Update the value to be the fully qualified domain name (FQDN) or IP address of the IoT Edge device. Use whichever value you choose consistently across the devices in your hierarchy.

    hostname: <device fqdn or IP>
    
  6. For IoT Edge devices in lower layers, update the config file to point to the FQDN or IP of the parent device, matching whatever is in the parent device's hostname field. For IoT Edge devices in the top layer, leave this parameter commented out.

    parent_hostname: <parent device fqdn or IP>
    

    Note

    For hierarchies with more than one lower layer, update the parent_hostname field with the FQDN of the device in the layer immediately above.

  7. For your top layer device, find the agent yaml section and update the image value to the correct version of the IoT Edge agent. In this case, we will point the top layer's IoT Edge agent at the Azure Container Registry with the public preview version of IoT Edge agent image available.

    agent:
      name: "edgeAgent"
      type: "docker"
      env: {}
      config:
        image: "mcr.microsoft.com/azureiotedge-agent:1.2.0-rc2"
        auth: {}
    
  8. For IoT Edge devices in lower layers, update the image value's domain name to point to the parent device's FQDN or IP followed by the API proxy port, 8000. You will add the API proxy module in the next section.

    agent:
      name: "edgeAgent"
      type: "docker"
      env: {}
      config:
        image: "<parent_device_fqdn_or_ip>:8000/azureiotedge-agent:1.2.0-rc2"
        auth: {}
    
  9. Save and close the file.

    CTRL + X, Y, Enter

  10. After entering the provisioning information in the configuration file, restart the daemon:

    sudo systemctl restart iotedge
    

Deploy modules to the top layer device

The remaining steps to complete the configuration of the IoT Edge runtime and deploy workloads are done from the Cloud either via the Azure portal or Azure CLI.

In the Azure portal:

  1. Navigate to your IoT Hub.

  2. From the menu on the left pane, under Automatic Device Management, select IoT Edge.

  3. Click on the device ID of your top layer edge device in the list of devices.

  4. On the upper bar, select Set Modules.

  5. Select Runtime Settings, next to the gear icon.

  6. Under Edge Hub, in the image field, enter mcr.microsoft.com/azureiotedge-hub:1.2.0-rc2.

    Edit the Edge Hub's image

  7. Add the following environment variables to your Edge Hub module:

    Name Value
    experimentalFeatures__enabled true
    experimentalFeatures__nestedEdgeEnabled true

    Edit the Edge Hub's environment variables

  8. Under Edge Agent, in the image field, enter mcr.microsoft.com/azureiotedge-agent:1.2.0-rc2. Select Save.

  9. Add the Docker registry module to your top layer device. Select + Add and choose IoT Edge Module from the dropdown. Provide the name registry for your Docker registry module and enter registry:latest for the image URI. Next, add environment variables and create options to point your local registry module at the Microsoft container registry to download container images from and to serve these images at registry:5000.

  10. Under the environment variables tab, enter the following environment variable name-value pair:

    Name Value
    REGISTRY_PROXY_REMOTEURL https://mcr.microsoft.com
  11. Under the container create options tab, enter the following JSON:

    {
     "HostConfig": {
         "PortBindings": {
             "5000/tcp": [
                 {
                     "HostPort": "5000"
                 }
             ]
          }
       }
    }
    
  12. Next, add the API proxy module to your top layer device. Select + Add and choose Marketplace Module from the dropdown. Search for IoT Edge API Proxy and select the module. The IoT Edge API proxy uses port 8000 and is configured to use your registry module named registry on port 5000 by default.

  13. Select Review + create, then Create to complete the deployment. Your top layer device's IoT Edge runtime, which has access to the internet, will pull and run the public preview configs for IoT Edge hub and IoT Edge agent.

    Complete deployment containing Edge Hub, Edge Agent, Registry Module, and API Proxy Module

Deploy modules to the lower layer device

You can use both the Azure portal and Azure CLI to deploy workloads from the Cloud to your lower layer devices.

In the Azure portal:

  1. Navigate to your IoT Hub.

  2. From the menu on the left pane, under Automatic Device Management, select IoT Edge.

  3. Click on the device ID of your lower layer device in the list of IoT Edge devices.

  4. On the upper bar, select Set Modules.

  5. Select Runtime Settings, next to the gear icon.

  6. Under Edge Hub, in the image field, enter $upstream:8000/azureiotedge-hub:1.2.0-rc2.

  7. Add the following environment variables to your Edge Hub module:

    Name Value
    experimentalFeatures__enabled true
    experimentalFeatures__nestedEdgeEnabled true
  8. Under Edge Agent, in the image field, enter $upstream:8000/azureiotedge-agent:1.2.0-rc2. Select Save.

  9. Add the temperature sensor module. Select + Add and choose Marketplace Module from the dropdown. Search for Simulated Temperature Sensor and select the module.

  10. Under IoT Edge Modules, select the Simulated Temperature Sensor module you just added and update its image URI to point to $upstream:8000/azureiotedge-simulated-temperature-sensor:1.0.

  11. Select Save, Review + create, and Create to complete the deployment.

    Complete deployment containing Edge Hub, Edge Agent, and Simulated Temperature Sensor

Notice that the image URI that we used for the simulated temperature sensor module pointed to $upstream:8000 instead of to a container registry. We configured this device to not have direct connections to the cloud, because it's in a lower layer. To pull container images, this device requests the image from its parent device instead. At the top layer, the API proxy module routes this container request to the registry module, which handles the image pull.

On the device details page for your lower layer IoT Edge device, you should now see the temperature sensor module listed along the system modules as Specified in deployment. It may take a few minutes for the device to receive its new deployment, request the container image, and start the module. Refresh the page until you see the temperature sensor module listed as Reported by device.

IotEdge check

Run the iotedge check command to verify the configuration and to troubleshoot errors.

You can run iotedge check in a nested hierarchy, even if the child machines don't have direct internet access.

When you run iotedge check from the lower layer, the program tries to pull the image from the parent through port 443.

In this tutorial, we use port 8000, so we need to specify it:

sudo iotedge check --diagnostics-image-name <parent_device_fqdn_or_ip>:8000/azureiotedge-diagnostics:1.2.0-rc2

The azureiotedge-diagnostics value is pulled from the container registry that's linked with the registry module. This tutorial has it set by default to https://mcr.microsoft.com:

Name Value
REGISTRY_PROXY_REMOTEURL https://mcr.microsoft.com

If you're using a private container registry, make sure that all the images (for example, IoTEdgeAPIProxy, edgeAgent, edgeHub, and diagnostics) are present in the container registry.

View generated data

The Simulated Temperature Sensor module that you pushed generates sample environment data. It sends messages that include ambient temperature and humidity, machine temperature and pressure, and a timestamp.

You can watch the messages arrive at your IoT hub by using the Azure IoT Hub extension for Visual Studio Code.

You can also view these messages through the Azure Cloud Shell:

az iot hub monitor-events -n <iothub_name> -d <lower-layer-device-name>

Clean up resources

You can delete the local configurations and the Azure resources that you created in this article to avoid charges.

To delete the resources:

  1. Sign in to the Azure portal and select Resource groups.

  2. Select the name of the resource group that contains your IoT Edge test resources.

  3. Review the list of resources contained in your resource group. If you want to delete all of them, you can select Delete resource group. If you want to delete only some of them, you can click into each resource to delete them individually.

Next steps

In this tutorial, you configured two IoT Edge devices as gateways and set one as the parent device of the other. Then, you demonstrated pulling a container image onto the child device through a gateway.

To see how Azure IoT Edge can create more solutions for your business, continue on to the other tutorials.