July 2015
Volume 30 Number 7
Azure Insider - Event Hubs for Analytics and Visualization, Part 3
By Bruno Terkaly | July 2015
If you’ve been following this series, you’ll finally see the realization of the main goal of the previous work in this final installment—to visualize data originating from Raspberry Pi devices in the field. This is the final installment of a three-part series around an Internet of Things (IoT) scenario. For the previous two articles, check the April issue (msdn.microsoft.com/magazine/dn948106) and June issue (msdn.microsoft.com/magazine/mt147243).
The high-level goals for this month’s column are quite simple. I’ll build a Node.js application that acts as a Web site and provides data to the mobile device, which will then display a bar chart of rainfall data. I need the Node.js Web server because it’s generally not advisable for a mobile device to directly access data stores.
Other technologies could have performed this function, such as ASP.NET Web API or Microsoft Azure API Apps. Most architects would advise you to filter, sort and parse data in the middle tier on a server, not on the mobile device itself. That’s because mobile devices have less computing power and you don’t want to send large amounts of information across the wire to the device. In short, let the server do the heavy lifting and let the device display visuals.
Get Started
To run the examples in this month’s column, you’ll need to have completed the previous two from the April and June issues. If you haven’t, you can probably get by with manually building out your own DocumentDB data store.
Assuming you’ve previously created a DocumentDB data store, start by going to the Azure Portal and viewing the DocumentDB database. There’s some nice tooling at the portal, which lets you view your data and perform different queries. Once you validate the data is there and understand its basic structure, you can begin to build the Node.js Web server.
Developers generally like to validate software as they build it out. A lot of developers start with unit testing, for example, with tools such as Mocha or Jasmine. For example, before building the mobile client, it makes sense to ensure the Node.js Web server operates as expected. One approach is to use a Web proxy tool, such as Fiddler. That makes it easy to issue Web requests (such as HTTP GET) and view the response in native JSON format. This can be useful, because when building the mobile client, you can be confident any problems are related to the mobile client, not the Web service.
To keep things simple, use a Windows Phone device as the client, even though iOS and Android devices are more prevalent. Perhaps the best approach would be a Web-based technology. This would let you consume or visualize the data from any device, therefore not limiting yourself to mobile devices. Another approach is to use native cross-platform products, such as Xamarin.
There are several sources of information you can use to build out a Web-based mobile client, not the least of which is the W3C standards body. You can read more about them in the standards documents at bit.ly/1PQ6uOt. The Apache Software Foundation also has done a bunch of work in this space. Read more about that at cordova.apache.org. I focused on Windows Phone here because the code is straightforward, easy to debug and simple to explain.
DocumentDB at the Portal
In this next section, I’ll build on the previous work, wherein I created a DocumentDB data store for the city and temperature data. To find your DocumentDB database (TemperatureDB), simply click on the menu item, Browse, then select DocumentDB Accounts. You could also pin your database as a tile on the main page by right-clicking, making the main Azure Portal page a dashboard. You can find a nice summary of using the portal tooling to interact with your DocumentDB data store at bit.ly/112L4X1.
One of the really useful features at the new portal is the ability to query and view the actual data in DocumentDB, allowing you to see the city and temperature data in its native JSON format. This dramatically simplifies your ability to easily alter the Node.js Web server and to build out any necessary queries.
Build Out the Node.js Web Server
One of the first things you’ll need to do when creating your Node.js Web server is connect to the DocumentDB data store. You’ll find the needed connection strings on the Azure Portal. To get to the connection strings from the portal, select the TemperatureDB DocumentDB store, then Click on All Settings, followed by Keys.
From there, you’ll need two pieces of information. The first is the URI to the DocumentDB data store. The second is the information security key (called the Primary Key at the portal), which lets you enforce secure access from the Node.js Web server. You’ll see the connection and database information in the following code:
{
"HOST" : "https://temperaturedb.documents.azure.com:443/",
"AUTH_KEY" : "secret key from the portal",
"DATABASE" : "TemperatureDB",
"COLLECTION" : "CityTempCollection"
}
The Node.js Web server will read the configuration information. The host field in your configuration file will be different, because all URIs are globally unique. The authorization key is also unique.
In previous articles, the temperature database was named TemperatureDB. Each database can have multiple collections, but in this case there’s just one called the CityTemperature collection. A collection is nothing more than a list of documents. In this data model, a single document is a city with 12 months of temperature data.
As you dive into the details of the code for the Node.js Web server, you can take advantage of the extensive ecosystem of add-on libraries for Node.js. You’ll use two libraries (called Node Packages) for this project. The first package is for DocumentDB functionality. The command to install the DocumentDB package is: npm install documentdb. The second package is for reading the configuration file: npm install nconf. These packages provide additional capabilities missing in the default installation of Node.js. You can find a more extensive tutorial about building out a Node.js application for DocumentDB in the Azure documentation at bit.ly/1E7j5Wg.
There are seven sections in the Node.js Web server, as shown in Figure 1. Section 1 covers connections to some of the installed packages so you’ll have access to them later in the code. Section 1 also defines the default port to which the mobile client will connect. When deployed to Azure, the port number is managed by Azure, hence the process.env.port construct.
Figure 1 The Built Out Node.js Web Server
// +-----------------------------+
// | Section 1 |
// +-----------------------------+
var http = require('http');
var port = process.env.port || 1337;
var DocumentDBClient = require('documentdb').DocumentClient;
var nconf = require('nconf');
// +-----------------------------+
// | Section 2 |
// +-----------------------------+
// Tell nconf which config file to use
nconf.env();
nconf.file({ file: 'config.json' });
// Read the configuration data
var host = nconf.get("HOST");
var authKey = nconf.get("AUTH_KEY");
var databaseId = nconf.get("DATABASE");
var collectionId = nconf.get("COLLECTION");
// +-----------------------------+
// | Section 3 |
// +-----------------------------+
var client = new DocumentDBClient(host, { masterKey: authKey });
// +-----------------------------+
// | Section 4 |
// +-----------------------------+
http.createServer(function (req, res) {
// Before you can query for Items in the document store, you need to ensure you
// have a database with a collection, then use the collection
// to read the documents.
readOrCreateDatabase(function (database) {
readOrCreateCollection(database, function (collection) {
// Perform a query to retrieve data and display
listItems(collection, function (items) {
var userString = JSON.stringify(items);
var headers = {
'Content-Type': 'application/json',
'Content-Length': userString.length
};
res.write(userString);
res.end();
});
});
});
}).listen(8124,'localhost'); // 8124 seemed to be the
// port number that worked
// from my development machine.
// +-----------------------------+
// | Section 5 |
// +-----------------------------+
// If the database does not exist, then create it, or return the database object.
// Use queryDatabases to check if a database with this name already exists. If you
// can't find one, then go ahead and use createDatabase to create a new database
// with the supplied identifier (from the configuration file) on the endpoint
// specified (also from the configuration file).
var readOrCreateDatabase = function (callback) {
client.queryDatabases('SELECT * FROM root r WHERE r.id="' + databaseId +
'"').toArray(function (err, results) {
console.log('readOrCreateDatabase');
if (err) {
// Some error occured, rethrow up
throw (err);
}
if (!err && results.length === 0) {
// No error occured, but there were no results returned,
// indicating no database exists matching the query.
client.createDatabase({ id: databaseId }, function (err, createdDatabase) {
console.log('client.createDatabase');
callback(createdDatabase);
});
} else {
// we found a database
console.log('found a database');
callback(results[0]);
}
});
};
// +-----------------------------+
// | Section 6 |
// +-----------------------------+
// If the collection does not exist for the database provided, create it,
// or return the collection object. As with readOrCreateDatabase, this method
// first tried to find a collection with the supplied identifier. If one exists,
// it is returned and if one does not exist it is created for you.
var readOrCreateCollection = function (database, callback) {
client.queryCollections(database._self, 'SELECT * FROM root r WHERE r.id="' +
collectionId + '"').toArray(function (err, results) {
console.log('readOrCreateCollection');
if (err) {
// Some error occured, rethrow up
throw (err);
}
if (!err && results.length === 0) {
// No error occured, but there were no results returned, indicating no
// collection exists in the provided database matching the query.
client.createCollection(database._self, { id: collectionId },
function (err, createdCollection) {
console.log('client.createCollection');
callback(createdCollection);
});
} else {
// Found a collection
console.log('found a collection');
callback(results[0]);
}
});
};
// +-----------------------------+
// | Section 7 |
// +-----------------------------+
// Query the provided collection for all non-complete items.
// Use queryDocuments to look for all documents in the collection that are
// not yet complete, or where completed = false. It uses the DocumentDB query
// grammar, which is based on ANSI - SQL to demonstrate this familiar, yet
// powerful querying capability.
var listItems = function (collection, callback) {
client.queryDocuments(collection._self, 'SELECT c.City,
c.Temperatures FROM c where c.id="WACO- TX"').toArray(function (err, docs) {
console.log('called listItems');
if (err) {
throw (err);
}
callback(docs);
});
}
Section 2 reads from the config.json file, which contains the connection information, including the database and document collection. It always makes sense to take string literals relating to connection information and place them separately in a configuration file.
Section 3 is the client connection object you’ll use to interact with DocumentDB. The connection is passed to the constructor for DocumentDBClient.
Section 4 represents the code executed once the mobile client connects to the Node.js Web Server application. The createServer is a core primitive for Node.JS applications, and involves a number of concepts around the event loop and processing HTTP requests. You can read more about that construct (bit.ly/1FcNq1E).
This represents the high-level entry point for clients connecting to the Node.js Web server. It’s also responsible for calling other pieces of Node.js code that retrieves JSON data from DocumentDB. Once the data is retrieved, it will package it as a JSON-based payload, and deliver it to a mobile client. It leverages the request and response objects, which are parameters to the createServer function, (http.createServer(function (req, res)...).
Section 5 begins the DocumentDB query processing. Your DocumentDB data store can contain multiple databases. The purpose of Section 5 is to narrow down the data at the DocumentDB URI and point to a specific database. In this case, that’s TemperatureDB. You’ll also see some extra code that isn't directly used, but is there purely for educational purposes. You’ll also notice some code to create a database if one does not exist. Much of the logic in Section 5 and beyond is based on the DocumentDB npm package installed earlier.
Section 6 represents the next step in the data retrieval process. The code here is automatically called as a result of the code in Section 5. Section 6 further narrows the data, drilling down to the document collection using the database established (TemperatureDB) in Section 5. You’ll note the select statement that includes a where clause for the CityTemperature collection. That includes some code to create a collection if one does not exist.
Section 7 represents the final query executed before data is returned to the mobile client. To keep things simple, the query is hardcoded to return temperature data for the city of Waco, Texas. In a realistic scenario, the mobile client would pass in the city (based on user input or possibly the device’s location). The Node.js Web server would then parse the city passed in and append it to the Where clause in Section 7.
The Node.js Web server is now complete and ready to execute. Once it’s up and running it, it will wait indefinitely for client requests from the mobile device. You’ll run the Node.js Web server locally on your development machine. At this point, it makes sense to use Fiddler to start testing the Node.js Web server. Fiddler lets you issue HTTP requests (in this case, GET) to the Web service and view the response. Validating behavior in Fiddler can help you resolve any issues before building out the mobile client.
Now you’re ready to build out the mobile client, which involves two basic architectural components—the XAML UI and the CS Code-Behind (where the programming logic lives). The code shown in Figure 2 represents the markup for the visual interface on the mobile client.
Figure 2 The XAML Markup for Mobile Client Main Screen Bar Chart
<Page
x:Class="CityTempApp.MainPage"
xmlns="https://schemas.microsoft.com/winfx/2006/xaml/presentation"
xmlns:x="https://schemas.microsoft.com/winfx/2006/xaml"
xmlns:local="using:CityTempApp"
xmlns:d="https://schemas.microsoft.com/expression/blend/2008"
xmlns:mc="https://schemas.openxmlformats.org/markup-compatibility/2006"
mc:Ignorable="d"
xmlns:charting="using:WinRTXamlToolkit.Controls.DataVisualization.Charting"
Background="{ThemeResource ApplicationPageBackgroundThemeBrush}">
<Grid>
<!-- WinRTXamlToolkit chart control -->
<charting:Chart
x:Name="BarChart"
Title=""
Margin="5,0">
<charting:BarSeries
Title="Rain (in)"
IndependentValueBinding="{Binding Name}"
DependentValueBinding="{Binding Value}"
IsSelectionEnabled="True"/>
</charting:Chart>
</Grid>
</Page>
Notice that it incorporates the WinRTXamlToolkit, which you can find on CodePlex at bit.ly/1PQdXwO. This toolkit includes a number of interesting controls. The one you’ll use is the chart control. To chart data, simply build up a name/value collection and attach it to the control. In this case, you’ll build out a name/value collection of rainfall data for each month in a given city.
Before presenting the final solution, there are some caveats. One could dispute using a native Windows Phone application, in favor of a more Web-based approach.
The mobile client built out here takes a number of shortcuts to come up with the bare minimum functionality. For example, the bar chart will appear immediately once you run the Windows Phone client. That’s because there’s a Web request to the Node.js Web server from the OnNavigatedTo event. This automatically executes once the Windows Phone client starts. You can see this in Section 1 of the mobile client code shown in Figure 3.
Figure 3 Code for the Mobile Client
public sealed partial class MainPage : Page
{
public MainPage()
{
this.InitializeComponent();
this.NavigationCacheMode = NavigationCacheMode.Required;
}
// +-----------------------------+
// | Section 1 |
// +-----------------------------+
protected override void OnNavigatedTo(NavigationEventArgs e)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(
"https://localhost:8124");
request.BeginGetResponse(MyCallBack, request);
}
// +-----------------------------+
// | Section 2 |
// +-----------------------------+
async void MyCallBack(IAsyncResult result)
{
HttpWebRequest request = result.AsyncState as HttpWebRequest;
if (request != null)
{
try
{
WebResponse response = request.EndGetResponse(result);
Stream stream = response.GetResponseStream();
StreamReader reader = new StreamReader(stream);
JsonSerializer serializer = new JsonSerializer();
// +-----------------------------+
// | Section 3 |
// +-----------------------------+
// Data structures coming back from Node
List<CityTemp> cityTemp = (List<CityTemp>)serializer.Deserialize(
reader, typeof(List<CityTemp>));
// Data structure suitable for the chart control on the phone
List<NameValueItem> items = new List<NameValueItem>();
string[] months = { "Jan", "Feb", "Mar", "Apr", "May", "Jun",
"Jul", "Aug", "Sep", "Oct", "Nov", "Dec" };
for (int i = 11; i >= 0; i-- )
{
items.Add(new NameValueItem { Name = months[i], Value =
cityTemp[0].Temperatures[i] });
}
// +-----------------------------+
// | Section 4 |
// +-----------------------------+
// Provide bar chart the data
await this.Dispatcher.RunAsync(CoreDispatcherPriority.Normal, () =>
{
this.BarChart.Title = cityTemp[0].City + ", 2014";
((BarSeries)this.BarChart.Series[0]).ItemsSource = items;
});
}
catch (WebException e)
{
return;
}
}
}
}
// +-----------------------------+
// | Section 5 |
// +-----------------------------+
// Data structures coming back from Node
public class CityTemp
{
public string City { get; set; }
public List<double> Temperatures { get; set; }
}
// Data structure suitable for the chart control on the phone
public class NameValueItem
{
public string Name { get; set; }
public double Value { get; set; }
}
Also in Section 1, you’ll notice it’s connecting to the Node.js Web server running on localhost. Obviously, you would specify a different endpoint if you’re hosting your Node.js Web server in the public cloud, such as Azure Web sites. After issuing the Web request, you set up the callback using BeginGetResponse. Web requests are asynchronous, so the code sets up a callback (MyCallBack). This leads to Section 2, where the data is retrieved, processed and loaded into a chart control.
The callback in Section 2 for the asynchronous Web request outlined in Section 1 processes the payload sent back by the Web service. The code processes the Web response, serializing the data, which is in JSON format. Section 3 is about transforming that JSON data into two separate data structures defined in Section 5. The goal is to create a list of name/value arrays or lists. The NameValueItem class is the structure the chart control needs.
Section 4 is about using the GUI thread to assign the name/value list to the chart control. You can see the item assignment in the items source collection. The await this.Dispatcher.RunAsync syntax leverages the GUI thread for updating visual controls. The code won’t work properly if you try to update the visual interface using the same thread as the one processing the data and making the Web request.
Now you can run the mobile client. You might be missing some of the temperature data, though, so all the bar controls might not appear.
Wrapping Up
This concludes the three-part series, where I set out to show an end-to-end IoT scenario—from data ingestion to persisting the data to visualizing the data. I started this series by ingesting the data using a C program running under Ubuntu, as an effort to simulate the code you’d need to run under a Raspberry Pi device. You can insert data captured from a temperature sensor into Azure Event Hubs. However, data stored here is ephemeral and you need to move it to a more permanent store.
That brought you to the second article, in which a background process takes the data from Event Hubs and moves it to both Azure SQL Database and DocumentDB. Then this final installment exposed this permanently stored data to mobile devices using a middle tier running Node.js.
There are many potential extensions and improvements that can be performed. For example, one area you might explore is the notion of machine learning and analytics. The bar chart visualization answers the basic question, “What happened?” A more interesting question might be, “What will happen?” In other words, can you then predict future rainfall? The ultimate question could be, “What should I do today based on future predictions?”
Bruno Terkaly is a principal software engineer at Microsoft with the objective of enabling development of industry-leading applications and services across devices. He’s responsible for driving the top cloud and mobile opportunities across the United States and beyond from a technology-enablement perspective. He helps partners bring their applications to market by providing architectural guidance and deep technical engagement during the ISV’s evaluation, development and deployment. Terkaly also works closely with the cloud and mobile engineering groups, providing feedback and influencing the roadmap.
Thanks to the following Microsoft technical experts for reviewing this article: Leland Holmquest, David Magnotti and Nick Trogh
David Magnotti (Microsoft), Leland Holmquest (Microsoft), Nick Trogh (Microsoft)
David Magnotti is a Premier Field Engineer with Microsoft, specializing in software development. He primarily supports Microsoft enterprise customers through building proof of concepts, conducting advisory services, and delivering workshops. He can often be found debugging crash dumps and analyzing windows performance traces.
Leland Holmquest is a Solution Manager for the Knowledge Management platform for Microsoft Services, longtime fan of MSDN Magazine and supporter of do’ers!
Nick Trogh is a Technical Evangelist at Microsoft, where he helps developers realize their software dreams on the Microsoft platform. You can find him blogging, tweeting or speaking at developer events.