February 2019

Volume 34 Number 2

[Machine Learning]

Machine Learning on the Edge: Azure ML and Azure IOT Edge Integration

By Ted Way | February 2019

The edge is defined as any compute platform that isn’t the cloud. It could range from Azure Stack (a cluster of machines that serves as mini-Azure), to a powerful gateway server, to a smaller device such as a Raspberry Pi or sensor. Even refrigerators and other appliances have enough processing power nowadays to be considered edge devices. The challenge is how to apply artificial intelligence (AI) and machine learning (ML) to data that can’t make it to the cloud, whether that’s due to data sovereignty, privacy, bandwidth or other issues.

James McCaffrey had a great article—“Machine Learning with IoT Devices on the Edge” (msdn.com/magazine/mt847186), in the July 2018 issue of MSDN Magazine on training ML models and using the Microsoft Embedded Learning Library (ELL) to deploy an optimized model to small edge devices. In this article, our focus is on enterprise tools and services to train your model, manage it and containerize it with Azure Machine Learning. With Azure IoT Edge, you’ll learn how to configure a data pipeline utilizing the model and deploying it to the edge device. We’ll walk through an example where the Custom Vision service can be used to train a model that’s deployed with Azure IoT Edge. Iterations of the model can be registered with Azure ML.

Snow Leopard Scenario

The Snow Leopard Trust (snowleopard.org) is dedicated to “better understand the endangered snow leopard, and to protect the cat in partnership with communities that share its habitat.” It does this by setting up cameras across 200 square miles to track which snow leopards are in the area. The cameras are deployed and then data is collected from the camera after a set time. These cameras are motion-activated, which can produce some surprises—such as the time years ago when researchers developed a roll of film only to find photos of a tree swaying in the wind.

With digital cameras, more images can be captured, but even then, only 5 percent of the images contain snow leopards. It could take a human researcher more than 200 hours to sift through the pictures to identify those containing snow leopards. A far better use of a researcher’s time would be to train an ML algorithm to identify snow leopards.

ML algorithms are said to “learn,” rather than be explicitly programmed. For example, if you were creating an app to predict the price of a house based on factors such as the number of bedrooms, square footage and other factors, you could explicitly program it by adding rules or looking it up in some database that already has the answer. If the user enters the number of bedrooms between two and four, and the house had a certain square footage, you would return a certain price.

Instead of programming that response, you could use a very simple equation, such as:

y = w1*x1 + w2*x2 + b

In this example, x1 is the number of bedrooms and x2 is the square footage. Based on your data where you have the last sale price of a number of houses, you can “train” the model by finding w1, w2 and b that fits the data best. After training, the ML model is static, in that if you give it the same input twice, it will respond with the same output each time. However, you could continue to evolve the algorithm by training it on new data and updating the algorithm, which is essentially finding new values for w1, w2 and b.

Deep learning is a subset of ML. Instead of a simple linear equation like the one just described, we use artificial neural networks or deep neural networks (DNNs) that consist of many neurons or nodes across many layers. The goal is to mimic the way our brains work. Each neuron performs a calculation based on data, such as parts of images, and if the result meets a certain threshold, the neuron is activated.

This input is then fed into the next layer. For example, let’s say we had a picture of a handwritten number, and we have two neurons. The first neuron looks at the top half of the picture, and the second looks at the bottom half. If the first neuron sees a circle, it activates. If the bottom one sees a vertical line, it activates. For the number eight, the first neuron will activate, but the second one will not. However, for the number nine, the first neuron will activate, and the second one will also activate. You can then take that information and conclude that it’s the number nine.

Azure Components

Before we go any further, it’s important to be aware of the components of both the Azure and ML stacks. These include:

Custom Vision Service: This service is part of the Cognitive Services platform and lets you easily train an AI model by uploading images and manually tagging them. The model can be exported to many platforms—such as iOS (CoreML), ONNX or TensorFlow—and saved in the .pb format, based on Protocol Buffers, or protobufs. Protobufs are described by Google as its “language-neutral, platform-neutral, extensible mechanism for serializing structured data—think XML, but smaller, faster and simpler” (developers.google.com/protocol-buffers).

Azure Machine Learning: This service is an end-to-end data science platform as a cloud service that you can use to develop and deploy ML models. Using the Azure Machine Learning service, you can track your models as you build, train, deploy and manage them, all at the broad scale that the cloud provides.

Azure IoT Hub and IoT Edge: Azure IoT Hub is used to connect, monitor and control billions of Internet of Things (IoT) assets. Azure IoT Edge is a platform that’s remotely managed from an IoT Hub. It enables analysis of data on devices instead of in the cloud by moving parts of your workload to the edge. Your devices can spend less time sending messages to the cloud, and they can react autonomously and quickly to changes.

Using these components the Snow Leopard Trust can deploy 30 motion-activated cameras, each attached to relatively inexpensive PCs. We’ll use Azure IoT Edge to deploy the snow leopard model to a PC that simulates one of the devices.

Image Classification on the Edge

Now let’s walk through the steps involved in provisioning the required Azure resources and for building the solution. We’ll provide more details about what’s provisioned and what’s happening in the code in the following section. To get started you’ll need:

• An Azure subscription. Don’t have one? Create a free account at azure.microsoft.com/free.

• Visual Studio Code (code.visualstudio.com).

• Ubuntu 16.04 Linux x64 VM to serve as your IoT Edge device.

• Azure resources, including an Azure Machine Learning workspace, an Azure IoT Hub and a CustomVision.ai account.

• Model deployment sample at aka.ms/visedge.

Creating an Azure ML workspace also creates a storage account and Azure Container Registry, which you can use to store your Docker images for deployment to IoT Edge devices. Start by downloading and installing VS Code Tools for AI from (aka.ms/vscodetoolsforai). After installation, click the Azure symbol in the VS Code activity bar on the left.

Now install the Azure ML SDK by opening the command palette using Ctrl+Shift+P and typing “Install Azure ML SDK.” In the integrated terminal window, specify the Python interpreter to use or you can hit Enter to use your default Python interpreter.

Next, create an Azure ML workspace by clicking the Azure icon in the Visual Studio Code activity bar. The Azure: Machine Learning sidebar appears and then you right-click your Azure subscription and select Create Workspace. In the list that appears, select an existing resource group from the list, or create a new one using the wizard in the command palette. In this example, we’re using “iot-camera-group.”

Now in the field, type a unique and clear name for your new workspace. Press enter and the new workspace will be created. It appears in the tree below the subscription name.

Let’s create an IoT Hub by opening VS Code and clicking on the Extensions button on the left pane (Ctrl+Shift+X). Search for “Azure IoT Toolkit” and install the extension. You’ll need a Linux PC to act as your edge device. You can use a VM on your PC, on Azure or anywhere else you have access.

Start by clicking the Explorer button on the activities bar on the left and find the “Azure IoT Hub Devices” entry. Click “More actions” (the three dots) and select “Create IoT Hub.”

Now choose a subscription, then choose the resource group (“iot-camera-group”) from earlier or create a new one. Next, choose a region and choose a tier. You may use the free F1 tier, but if an F1 tier IoT Hub already exists, then you’ll need to create an S1 tier. Finally, give the IoT Hub a name, such as “iot-camera-hub.”

It’s time to register an IoT Edge device by adding an entry to your IoT Hub. This will generate a connection string that you’ll enter in your IoT Edge device (in this case, your Linux VM) so it can connect to your IoT Hub.

To do this, first click the “More actions” menu (the three dots) on “Azure IoT Hub Devices” and choose “Create IoT Edge device.” Don’t click “Create Device.” Then enter a name for the edge device, such as camera-pc-01. Finally, copy the connectionString value to the entry you just created. You’ll need to enter this in your edge device.

In the future, you can click “Azure IoT Hub Devices” and then “Select IoT Hub.” You’ll have to sign into Azure, choose the subscription you provisioned the IoT Hub in, and then select your IoT Hub, but you can view all the connection strings associated with your devices.

Set up IoT Edge Device

The easiest way to simulate an edge device is to use a preconfigured VM (aka.ms/edgevm). When the VM starts, it installs the latest Azure IoT Edge runtime and dependencies. After you provision the VM, log into it and run this command on the command line with the connection string for the device you registered with IoT Edge (the connection string is the connectionString .value you copied earlier):

/etc/iotedge/configedge.sh "<your_iothub_edge_connection_string>"

If you want to set up your own Ubuntu 16.04 x64 VM as the edge device, check out this information: aka.ms/edgevm.

Image Classification Solution

Now it’s time to build the image classification solution. Clone the Git sample (aka.ms/visedge). We’ll use the Linux x64 PC as the edge machine. Follow these steps:

• Update the .env file with the values for your container registry and make sure that your Docker engine has access to it.

• Rename the deployment.template.json file to deployment.template.RPI.json.

• Rename the deployment.template.test-amd64.json file to deployment.template.json.

• Build the entire solution by right-clicking the file deployment.template.json and selecting Build IoT Edge Solution (this can take a while, especially to build numpy and pillow).

• Deploy the solution to your device by right-clicking the config/deployment.json file, selecting Create Deployment for IoT Edge device and choosing your targeted device.

• Monitor the messages being sent to the Cloud by right-clicking your device from the VS Code IoT Edge Extension and selecting Start Monitoring D2C Message.

For the x64 version of the solution, these steps deploy two modules:

Camera Capture: This captures the video stream from a USB camera and sends the frames to the image classification module. The result is sent to edgeHub, which can relay it as a message to IoT Hub.

Custom Vision: This Web service runs locally over HTTP and takes in an image and classifies it. It was built using the Custom Vision (customvision.ai) Web site. The model in the solution classifies whether an image contains an apple or a banana.

The Camera Capture module can send messages to the edgeHub using the Azure IoT Edge SDK. You can build your own modules as Docker containers and use the SDK to communicate with edgeHub.

For the Custom Vision Service you can work with the existing model in the solution, or you can go to customvision.ai and train your own model. To create your snow leopard classifier, upload pictures of snow leopards to the Custom Vision Service. You can also upload pictures of other animals you may find in the snow leopard habitat. Just tag the images and click the Train button to train the model. The model and label files are exported as .pb and .txt files, respectively, which you can use to replace what’s in the solution.

Custom Vision uses transfer learning, which takes an existing deep neural network and extracts features from your image. Then it trains a classifier on your data (in this case, the images you uploaded and the tags you assigned to them).

To illustrate transfer learning, imagine you want to train a bomb-sniffing dog—a German Shepherd, for example. If you take a 2-year-old German Shepherd, it won’t know anything about bombs. It does have the “infrastructure” to detect bombs, with its powerful sense of smell. You can start training your German Shepherd by having it sniff a bomb and telling it that this is a bomb, then give it another smell, such as that of a flower, and say that this is not a bomb. You repeat this for various types of bombs and other smells, and after a few months and lots of kibble, you have yourself a bomb-sniffing dog. Training the German Shepherd isn’t that difficult. Making a German Shepherd is the hard part.

DNNs such as ResNet 50, Inception v3, YOLO and VGG are essentially German Shepherds. They’re complex AI models that take in an image and extract features from it, much the way a German Shepherd extracts signals from a smell. Just like the number of bedrooms and the square footage of a house are features that can be used to predict a price, the features extracted are sent to a classifier. This classifier is what you need to train, which will then do the prediction of whether a snow leopard is in the image or not. The classifier can be a simple logistic regression, support vector machine or any other classifier. The beauty in transfer learning is the feature extraction with the DNN, which results in useful features that can be used to classify images very accurately.

Figure 1 provides the code to take an image and predict what’s in it using the AI model.

Figure 1 Prediction Code

from urllib.request import urlopen

import tensorflow as tf

from PIL import Image
import numpy as np
# import scipy
# from scipy import misc
import sys
import os  

filename = 'model.pb'
labels_filename = 'labels.txt'

network_input_size = 227
mean_values_b_g_r = (0,0,0)

size = (256, 256)
output_layer = 'loss:0'
input_node = 'Placeholder:0'

graph_def = tf.GraphDef()
labels = []

def initialize():
  print('Loading model...',end=''),
  with tf.gfile.FastGFile(filename, 'rb') as f:
    tf.import_graph_def(graph_def, name='')
  print('Loading labels...', end='')
  with open(labels_filename, 'rt') as lf:
    for l in lf:
      l = l[:-1]
  print(len(labels), 'found. Success!')
def crop_center(img,cropx,cropy):
  y,x,z = img.shape
  startx = x//2-(cropx//2)
  starty = y//2-(cropy//2)
  print('crop_center: ', x, 'x', y, 'to', cropx, 'x', cropy)
  return img[starty:starty+cropy,startx:startx+cropx]

def predict_url(imageUrl):
  print('Predicting from url: ',imageUrl)
  with urlopen(imageUrl) as testImage:
    # image = scipy.misc.imread(testImage)
    image = Image.open(testImage)
    return predict_image(image)

def predict_image(image):
  print('Predicting image')
  tf.import_graph_def(graph_def, name='')
  with tf.Session() as sess:
    prob_tensor = sess.graph.get_tensor_by_name(output_layer)

    # w = image.shape[0]
    # h = image.shape[1]
    w, h = image.size
    print('Image size',w,'x',h)

    # scaling
    if w > h:
      new_size = (int((float(size[1]) / h) * w), size[1], 3)
      new_size = (size[0], int((float(size[0]) / w) * h), 3)

    # resize
    if  not (new_size[0] == w and new_size[0] == h):
      print('Resizing to', new_size[0],'x',new_size[1])
      #augmented_image = scipy.misc.imresize(image, new_size)
      augmented_image = np.asarray(image.resize((new_size[0], new_size[1])))
      augmented_image = np.asarray(image)

    # crop center
      augmented_image = crop_center(augmented_image, network_input_size, 
      return 'error: crop_center'

    augmented_image = augmented_image.astype(float)

    # RGB -> BGR
    red, green, blue = tf.split(axis=2, num_or_size_splits=3, value=augmented_image)

    image_normalized = tf.concat(axis=2, values=[
      blue - mean_values_b_g_r[0],
      green - mean_values_b_g_r[1],
      red - mean_values_b_g_r[2],

    image_normalized = image_normalized.eval()
    image_normalized = np.expand_dims(image_normalized, axis=0)

    predictions, = sess.run(prob_tensor, {input_node: image_normalized})

    result = []
    idx = 0
    for p in predictions:
      truncated_probablity = np.float64(round(p,8))
      if (truncated_probablity > 1e-8):
        result.append({'Tag': labels[idx], 'Probability': truncated_probablity })
      idx += 1
    print('Results: ', str(result))
    return result

Azure Machine Learning

You can use Custom Vision to create your AI model, but what if you want to train multiple models or collaborate with a team of data scientists? The rigors of software development, especially around source control, are missing in the data science workflow. Azure ML brings this all together with data preparation, experimentation, model management and operationalization.

For this example, if you’re doing various experiments with Custom Vision, you can register the .pb model and other files so you can keep track of them. Here’s that code:

from azureml.core.model import Model
model = Model.register(model_path = "custom-vision-model/",
                       model_name = "model.pb",
                       tags = {"area": "vision", "type": "classification", 
                         "version": "1.0"},
                       description = "Snow leopard classification model",
                       workspace = ws)

If you want to see all the models you’ve registered, enter this code:

models = Model.list(workspace=ws)
for m in models:
  print("Name:", m.name,"\tVersion:", m.version, "\tDescription:", 
  m.description, m.tags)

If you want to train your own models, check out the Azure Machine Learning sample notebooks (github.com/Azure/MachineLearningNotebooks). You can train your model, register it and create Docker images with the model. The Docker images can be deployed to the cloud in an Azure Container Instance or Azure Kubernetes Service. A REST API is exposed in the container to call when scoring data. The Docker image also contains the Azure IoT Edge SDK, which will broker messages with the IoT Edge agent running on the edge device. This lets you create the container and deploy it to the cloud or edge.

Wrapping Up

There are a lot of components when building an enterprise solution, starting with Custom Vision to train the model, Azure ML to manage the model (and to train other types of models) and Azure IoT Edge to deploy the models. Setting up these services will take some ramping up time, but you’ll reap the benefits of easily training new models and updating them on multiple edge devices.

Ted Wa is a senior program manager on the Azure ML team working to accelerate AI in the cloud and the edge on specialized hardware. His Ph.D. from the Uni-versity of Michigan was on “spell check for radiologists,” a computer-aided diagnosis system for estimating malignancy on thoracic CT scans.

Emmanuel Bertrand is a senior program manager on the Azure IoT Edge team, where he helps enable easy deployment of IoT modules through the Azure Marketplace. Ber-trand joined Microsoft as a customer and user experience expert, and prior to that had digitized the processes of several industrial companies using tradi-tional tools.

Thanks to the following technical experts for reviewing this article: Rakesh Kelkar, Chipalo Street

Discuss this article in the MSDN Magazine forum