Analyze real-time Twitter sentiment with HBase in HDInsight

Learn how to do real-time sentiment analysis of big data from Twitter by using a HBase cluster in HDInsight.

Social websites are one of the major driving forces for big data adoption. Public APIs provided by sites like Twitter are a useful source of data for analyzing and understanding popular trends. In this tutorial, you develop a console streaming service application and an ASP.NET web application to perform the following:

HDInsight HBase Analyze Twitter sentiment

  • The streaming application

    • get geo-tagged tweets in real time by using the Twitter streaming API
    • evaluate the sentiment of these tweets
    • store the sentiment information in HBase by using the Microsoft HBase SDK
  • The Azure Websites application

    • plot the real-time statistical results on Bing maps by using an ASP.NET web application. A visualization of the tweets is similar to the following screenshot:

      hdinsight.hbase.twitter.sentiment.bing.map

      You are able to query tweets with certain keywords to get a sense of if the expressed opinion in the tweets is positive, negative, or neutral.

A complete Visual Studio solution sample can be found on GitHub: Realtime social sentiment analysis app.

Prerequisites

Before you begin this tutorial, you must have the following:

Create a Twitter application ID and secrets

The Twitter streaming APIs use OAuth to authorize requests. The first step to use OAuth is to create a new application on the Twitter developer site.

To create Twitter application ID and secrets

  1. Sign in to Twitter Apps. Click the Sign up now link if you don't have a Twitter account.
  2. Click Create New App.
  3. Enter a Name, Description, and Website. The Twitter application name must be a unique name. The Website field is not really used. It doesn't have to be a valid URL.
  4. Check Yes, I agree, and then click Create your Twitter application.
  5. Click the Permissions tab, and then click Read only. The read-only permission is sufficient for this tutorial.
  6. Click the Keys and Access Tokens tab.
  7. Click Create my access token on the bottom of the page.
  8. Copy the Consumer Key (API Key), Consumer Secret (API Secret), Access token, and Access token secret values. You need these values later in the tutorial.

    ![NOTE] The Test OAuth button does not work anymore.

Create Twitter streaming service

You need to create an application to get tweets, calculate tweet sentiment score, and send the processed tweet words to HBase.

To create the streaming application

  1. Open Visual Studio, and create a Visual C# console application called TweetSentimentStreaming.
  2. From Package Manager Console, run the following commands:

     Install-Package Microsoft.HBase.Client -version 0.4.2.0
     Install-Package TweetinviAPI -version 1.0.0.0
    

    These commands install the HBase .NET SDK package, which is the client library to access the HBase cluster, and the Tweetinvi API package, which is used to access the Twitter API.

    Note

    The sample used in this article has been tested using the version specified above. You can remove the -version switch to install the latest version.

  3. From Solution Explorer, add System.Configuration to the reference.
  4. Add a new class file to the project called HBaseWriter.cs, and then replace the code with the following:

     using System;
     using System.Collections.Generic;
     using System.IO;
     using System.Linq;
     using System.Text;
     using System.Threading;
     using Microsoft.Practices.EnterpriseLibrary.TransientFaultHandling;
     using org.apache.hadoop.hbase.rest.protobuf.generated;
     using Microsoft.HBase.Client;
     using Tweetinvi.Models;
    
     namespace TweetSentimentStreaming
     {
         class HBaseWriter
         {
             // HDinsight HBase cluster and HBase table information
             const string CLUSTERNAME = "https://<Enter Your Cluster Name>.azurehdinsight.net/";
             const string HADOOPUSERNAME = "admin"; //the default name is "admin"
             const string HADOOPUSERPASSWORD = "<Enter the Hadoop User Password>";
    
             const string HBASETABLENAME = "tweets_by_words";
             const string COUNT_ROW_KEY = "~ROWCOUNT";
             const string COUNT_COLUMN_NAME = "d:COUNT";
    
             long rowCount = 0;
    
             // Sentiment dictionary file and the punctuation characters
             const string DICTIONARYFILENAME = @"..\..\dictionary.tsv";
             private static char[] _punctuationChars = new[] {
         ' ', '!', '\"', '#', '$', '%', '&', '\'', '(', ')', '*', '+', ',', '-', '.', '/',   //ascii 23--47
         ':', ';', '<', '=', '>', '?', '@', '[', ']', '^', '_', '`', '{', '|', '}', '~' };   //ascii 58--64 + misc.
    
             // For writting to HBase
             HBaseClient client;
    
             // a sentiment dictionary for estimate sentiment. It is loaded from a physical file.
             Dictionary<string, DictionaryItem> dictionary;
    
             // use multithread write
             Thread writerThread;
             Queue<ITweet> queue = new Queue<ITweet>();
             bool threadRunning = true;
    
             // This function connects to HBase, loads the sentiment dictionary, and starts the thread for writting.
             public HBaseWriter()
             {
                 ClusterCredentials credentials = new ClusterCredentials(new Uri(CLUSTERNAME), HADOOPUSERNAME, HADOOPUSERPASSWORD);
                 client = new HBaseClient(credentials);
    
                 // create the HBase table if it doesn't exist
                 if (!client.ListTablesAsync().Result.name.Contains(HBASETABLENAME))
                 {
                     TableSchema tableSchema = new TableSchema();
                     tableSchema.name = HBASETABLENAME;
                     tableSchema.columns.Add(new ColumnSchema { name = "d" });
                     client.CreateTableAsync(tableSchema).Wait();
                     Console.WriteLine("Table \"{0}\" is created.", HBASETABLENAME);
                 }
    
                 // Read current row count cell
                 rowCount = GetRowCount();
    
                 // Load sentiment dictionary from a file
                 LoadDictionary();
    
                 // Start a thread for writting to HBase
                 writerThread = new Thread(new ThreadStart(WriterThreadFunction));
                 writerThread.Start();
             }
    
             ~HBaseWriter()
             {
                 threadRunning = false;
             }
    
             private long GetRowCount()
             {
                 try
                 {
                     RequestOptions options = RequestOptions.GetDefaultOptions();
                     options.RetryPolicy = RetryPolicy.NoRetry;
                     var cellSet = client.GetCellsAsync(HBASETABLENAME, COUNT_ROW_KEY, null, null, options).Result;
                     if (cellSet.rows.Count != 0)
                     {
                         var countCol = cellSet.rows[0].values.Find(cell => Encoding.UTF8.GetString(cell.column) == COUNT_COLUMN_NAME);
                         if (countCol != null)
                         {
                             return Convert.ToInt64(Encoding.UTF8.GetString(countCol.data));
                         }
                     }
                 }
                 catch(Exception ex)
                 {
                     if (ex.InnerException.Message.Equals("The remote server returned an error: (404) Not Found.", StringComparison.OrdinalIgnoreCase))
                     {
                         return 0;
                     }
                     else
                     {
                         throw ex;
                     }
    
                 }
    
                 return 0;
             }
    
             // Enqueue the Tweets received
             public void WriteTweet(ITweet tweet)
             {
                 lock (queue)
                 {
                     queue.Enqueue(tweet);
                 }
             }
    
             // Load sentiment dictionary from a file
             private void LoadDictionary()
             {
                 List<string> lines = File.ReadAllLines(DICTIONARYFILENAME).ToList();
                 var items = lines.Select(line =>
                 {
                     var fields = line.Split('\t');
                     var pos = 0;
                     return new DictionaryItem
                     {
                         Type = fields[pos++],
                         Length = Convert.ToInt32(fields[pos++]),
                         Word = fields[pos++],
                         Pos = fields[pos++],
                         Stemmed = fields[pos++],
                         Polarity = fields[pos++]
                     };
                 });
    
                 dictionary = new Dictionary<string, DictionaryItem>();
                 foreach (var item in items)
                 {
                     if (!dictionary.Keys.Contains(item.Word))
                     {
                         dictionary.Add(item.Word, item);
                     }
                 }
             }
    
             // Calculate sentiment score
             private int CalcSentimentScore(string[] words)
             {
                 Int32 total = 0;
                 foreach (string word in words)
                 {
                     if (dictionary.Keys.Contains(word))
                     {
                         switch (dictionary[word].Polarity)
                         {
                             case "negative": total -= 1; break;
                             case "positive": total += 1; break;
                         }
                     }
                 }
                 if (total > 0)
                 {
                     return 1;
                 }
                 else if (total < 0)
                 {
                     return -1;
                 }
                 else
                 {
                     return 0;
                 }
             }
    
             // Popular a CellSet object to be written into HBase
             private void CreateTweetByWordsCells(CellSet set, ITweet tweet)
             {
                 // Split the Tweet into words
                 string[] words = tweet.Text.ToLower().Split(_punctuationChars);
    
                 // Calculate sentiment score base on the words
                 int sentimentScore = CalcSentimentScore(words);
                 var word_pairs = words.Take(words.Length - 1)
                                     .Select((word, idx) => string.Format("{0} {1}", word, words[idx + 1]));
                 var all_words = words.Concat(word_pairs).ToList();
    
                 // For each word in the Tweet add a row to the HBase table
                 foreach (string word in all_words)
                 {
                     string time_index = (ulong.MaxValue - (ulong)tweet.CreatedAt.ToBinary()).ToString().PadLeft(20) + tweet.IdStr;
                     string key = word + "_" + time_index;
    
                     // Create a row
                     var row = new CellSet.Row { key = Encoding.UTF8.GetBytes(key) };
    
                     // Add columns to the row, including Tweet identifier, language, coordinator(if available), and sentiment
                     var value = new Cell { column = Encoding.UTF8.GetBytes("d:id_str"), data = Encoding.UTF8.GetBytes(tweet.IdStr) };
                     row.values.Add(value);
    
                     value = new Cell { column = Encoding.UTF8.GetBytes("d:lang"), data = Encoding.UTF8.GetBytes(tweet.Language.ToString()) };
                     row.values.Add(value);
    
                     if (tweet.Coordinates != null)
                     {
                         var str = tweet.Coordinates.Longitude.ToString() + "," + tweet.Coordinates.Latitude.ToString();
                         value = new Cell { column = Encoding.UTF8.GetBytes("d:coor"), data = Encoding.UTF8.GetBytes(str) };
                         row.values.Add(value);
                     }
    
                     value = new Cell { column = Encoding.UTF8.GetBytes("d:sentiment"), data = Encoding.UTF8.GetBytes(sentimentScore.ToString()) };
                     row.values.Add(value);
    
                     set.rows.Add(row);
                 }
             }
    
             // Write a Tweet (CellSet) to HBase
             public void WriterThreadFunction()
             {
                 try
                 {
                     while (threadRunning)
                     {
                         if (queue.Count > 0)
                         {
                             CellSet set = new CellSet();
                             lock (queue)
                             {
                                 do
                                 {
                                     ITweet tweet = queue.Dequeue();
                                     CreateTweetByWordsCells(set, tweet);
                                 } while (queue.Count > 0);
                             }
    
                             // Write the Tweet by words cell set to the HBase table
                             client.StoreCellsAsync(HBASETABLENAME, set).Wait();
                             Console.WriteLine("\tRows written: {0}", set.rows.Count);
                         }
                         Thread.Sleep(100);
                     }
                 }
                 catch (Exception ex)
                 {
                     Console.WriteLine("Exception: " + ex.Message);
                 }
             }
         }
         public class DictionaryItem
         {
             public string Type { get; set; }
             public int Length { get; set; }
             public string Word { get; set; }
             public string Pos { get; set; }
             public string Stemmed { get; set; }
             public string Polarity { get; set; }
         }
     }
    
  5. Set the constants in the previous code, including CLUSTERNAME, HADOOPUSERNAME, HADOOPUSERPASSWORD, and DICTIONARYFILENAME. The DICTIONARYFILENAME is the filename and the location of the direction.tsv. The file can be downloaded from https://hditutorialdata.blob.core.windows.net/twittersentiment/dictionary.tsv. If you want to change the HBase table name, you must change the table name in the web application accordingly.
  6. Open Program.cs, and replace the code with the following:

     using System;
     using System.Diagnostics;
     using Tweetinvi;
     using Tweetinvi.Models;
    
     namespace TweetSentimentStreaming
     {
         class Program
         {
             const string TWITTERAPPACCESSTOKEN = "<Enter Twitter App Access Token>";
             const string TWITTERAPPACCESSTOKENSECRET = "<Enter Twitter Access Token Secret>";
             const string TWITTERAPPAPIKEY = "<Enter Twitter App API Key>";
             const string TWITTERAPPAPISECRET = "<Enter Twitter App API Secret>";
    
             static void Main(string[] args)
             {
                 Auth.SetUserCredentials(TWITTERAPPAPIKEY, TWITTERAPPAPISECRET, TWITTERAPPACCESSTOKEN, TWITTERAPPACCESSTOKENSECRET);
    
                 Stream_FilteredStreamExample();
             }
    
             private static void Stream_FilteredStreamExample()
             {
                 for (;;)
                 {
                     try
                     {
                         HBaseWriter hbase = new HBaseWriter();
                         var stream = Stream.CreateFilteredStream();
                         stream.AddLocation(new Coordinates(90, -180), new Coordinates(-90,180));
    
                         var tweetCount = 0;
                         var timer = Stopwatch.StartNew();
    
                         stream.MatchingTweetReceived += (sender, args) =>
                         {
                             tweetCount++;
                             var tweet = args.Tweet;
    
                             // Write Tweets to HBase
                             hbase.WriteTweet(tweet);
    
                             if (timer.ElapsedMilliseconds > 1000)
                             {
                                 if (tweet.Coordinates != null)
                                 {
                                     Console.ForegroundColor = ConsoleColor.Green;
                                     Console.WriteLine("\n{0}: {1} {2}", tweet.Id, tweet.Language.ToString(), tweet.Text);
                                     Console.ForegroundColor = ConsoleColor.White;
                                     Console.WriteLine("\tLocation: {0}, {1}", tweet.Coordinates.Longitude, tweet.Coordinates.Latitude);
                                 }
    
                                 timer.Restart();
                                 Console.WriteLine("\tTweets/sec: {0}", tweetCount);
                                 tweetCount = 0;
                             }
                         };
    
                         stream.StartStreamMatchingAllConditions();
                     }
                     catch (Exception ex)
                     {
                         Console.WriteLine("Exception: {0}", ex.Message);
                     }
                 }
             }
    
         }
     }
    
  7. Set the constants including TWITTERAPPACCESSTOKEN, TWITTERAPPACCESSTOKENSECRET, TWITTERAPPAPIKEY and TWITTERAPPAPISECRET.

To run the streaming service, press F5. The following is a screenshot of the console application:

hdinsight.hbase.twitter.sentiment.streaming.service

Keep the streaming console application running while you develop the web application, so you have more data to use. To examine the data inserted into the table, you can use HBase Shell. See Get started with HBase in HDInsight.

Visualize real-time sentiment

In this section, you create an ASP.NET MVC web application to read the real-time sentiment data from HBase and plot the data on Bing maps.

To create an ASP.NET MVC Web application

  1. Open Visual Studio.
  2. Click File, click New, and then click Project.
  3. Enter the following information:

    • Template category: Visual C#/Web
    • Template: ASP.NET Web Application
    • Name: TweetSentimentWeb
    • Location: C:\Tutorials
  4. Click OK.
  5. In Select a template, click MVC.
  6. In Microsoft Azure, click Manage Subscriptions.
  7. From Manage Microsoft Azure Subscriptions, click Sign in.
  8. Enter your Azure credentials. Your Azure subscription information is shown on the Accounts tab.
  9. Click Close to close the Manage Microsoft Azure Subscriptions window.
  10. From New ASP.NET Project - TweetSentimentWeb, click OK.
  11. From Configure Microsoft Azure Site Settings, select the Region that is closest to you. You don't need to specify a database server.
  12. Click OK.

To install NuGet packages

  1. From the Tools menu, click Nuget Package Manager, and then click Package Manager Console. The console panel is opened at the bottom of the page.
  2. Use the following command to install the HBase .NET SDK package, which is the client library to access HBase cluster:

     Install-Package Microsoft.HBase.Client
    

To add HBaseReader class

  1. From Solution Explorer, expand TweetSentiment.
  2. Right-click Models, click Add, and then click Class.
  3. In the Name field, type HBaseReader.cs, and then click Add.
  4. Replace the code with the following:

     using System;
     using System.Collections.Generic;
     using System.Linq;
     using System.Web;
    
     using System.Configuration;
     using System.Threading.Tasks;
     using System.Text;
     using Microsoft.HBase.Client;
     using org.apache.hadoop.hbase.rest.protobuf.generated;
    
     namespace TweetSentimentWeb.Models
     {
         public class HBaseReader
         {
             // For reading Tweet sentiment data from HDInsight HBase
             HBaseClient client;
    
             // HDinsight HBase cluster and HBase table information
             const string CLUSTERNAME = "<HBaseClusterName>";
             const string HADOOPUSERNAME = "<HBaseClusterHadoopUserName>"
             const string HADOOPUSERPASSWORD = "<HBaseCluserUserPassword>";
             const string HBASETABLENAME = "tweets_by_words";
    
             // The constructor
             public HBaseReader()
             {
                 ClusterCredentials creds = new ClusterCredentials(
                                 new Uri(CLUSTERNAME),
                                 HADOOPUSERNAME,
                                 HADOOPUSERPASSWORD);
                 client = new HBaseClient(creds);
             }
    
             // Query Tweets sentiment data from the HBase table asynchronously
             public async Task<IEnumerable<Tweet>> QueryTweetsByKeywordAsync(string keyword)
             {
                 List<Tweet> list = new List<Tweet>();
    
                 // Demonstrate Filtering the data from the past 6 hours the row key
                 string timeIndex = (ulong.MaxValue -
                     (ulong)DateTime.UtcNow.Subtract(new TimeSpan(6, 0, 0)).ToBinary()).ToString().PadLeft(20);
                 string startRow = keyword + "_" + timeIndex;
                 string endRow = keyword + "|";
                 Scanner scanSettings = new Scanner
                 {
                     batch = 100000,
                     startRow = Encoding.UTF8.GetBytes(startRow),
                     endRow = Encoding.UTF8.GetBytes(endRow)
                 };
    
                 // Make async scan call
                 ScannerInformation scannerInfo =
                     await client.CreateScannerAsync(HBASETABLENAME, scanSettings);
    
                 CellSet next;
    
                 while ((next = await client.ScannerGetNextAsync(scannerInfo)) != null)
                 {
                     foreach (CellSet.Row row in next.rows)
                     {
                         // find the cell with string pattern "d:coor"
                         var coordinates =
                             row.values.Find(c => Encoding.UTF8.GetString(c.column) == "d:coor");
    
                         if (coordinates != null)
                         {
                             string[] lonlat = Encoding.UTF8.GetString(coordinates.data).Split(',');
    
                             var sentimentField =
                                 row.values.Find(c => Encoding.UTF8.GetString(c.column) == "d:sentiment");
                             Int32 sentiment = 0;
                             if (sentimentField != null)
                             {
                                 sentiment = Convert.ToInt32(Encoding.UTF8.GetString(sentimentField.data));
                             }
    
                             list.Add(new Tweet
                             {
                                 Longtitude = Convert.ToDouble(lonlat[0]),
                                 Latitude = Convert.ToDouble(lonlat[1]),
                                 Sentiment = sentiment
                             });
                         }
    
                         if (coordinates != null)
                         {
                             string[] lonlat = Encoding.UTF8.GetString(coordinates.data).Split(',');
                         }
                     }
                 }
    
                 return list;
             }
         }
    
         public class Tweet
         {
             public string IdStr { get; set; }
             public string Text { get; set; }
             public string Lang { get; set; }
             public double Longtitude { get; set; }
             public double Latitude { get; set; }
             public int Sentiment { get; set; }
         }
     }
    
  5. Inside the HBaseReader class, change the constant values as follows:

    • CLUSTERNAME: The HBase cluster name, for example, https://.azurehdinsight.net/.
    • HADOOPUSERNAME: The HBase cluster Hadoop user user name. The default name is admin.
    • HADOOPUSERPASSWORD: The HBase cluster Hadoop user password.
    • HBASETABLENAME = "tweets_by_words";

      The HBase table name is "tweets_by_words";. The values must match the values you sent in the streaming service, so that the web application reads the data from the same HBase table.

To add TweetsController controller

  1. From Solution Explorer, expand TweetSentimentWeb.
  2. Right-click Controllers, click Add, and then click Controller.
  3. Click Web API 2 Controller - Empty, and then click Add.
  4. In the Controller name field, type TweetsController, and then click Add.
  5. From Solution Explorer, double-click TweetsController.cs to open the file.
  6. Modify the file, so it looks like the following:

     using System;
     using System.Collections.Generic;
     using System.Linq;
     using System.Net;
     using System.Net.Http;
     using System.Web.Http;
    
     using System.Threading.Tasks;
     using TweetSentimentWeb.Models;
    
     namespace TweetSentimentWeb.Controllers
     {
         public class TweetsController : ApiController
         {
             HBaseReader hbase = new HBaseReader();
    
             public async Task<IEnumerable<Tweet>> GetTweetsByQuery(string query)
             {
                 return await hbase.QueryTweetsByKeywordAsync(query);
             }
         }
     }
    

To add heatmap.js

  1. From Solution Explorer, expand TweetSentimentWeb.
  2. Right-click Scripts, click Add, click JavaScript File.
  3. In the Item name field, type heatmap.js.
  4. Paste the following code into the file. The code was written by Alastair Aitchison. For more information, see Bing Maps AJAX v7 HeatMap Library.

     /*******************************************************************************
     * Author: Alastair Aitchison
     * Website: http://alastaira.wordpress.com
     * Date: 15th April 2011
     *
     * Description:
     * This JavaScript file provides an algorithm that can be used to add a heatmap
     * overlay on a Bing Maps v7 control. The intensity and temperature palette
     * of the heatmap are designed to be easily customisable.
     *
     * Requirements:
     * The heatmap layer itself is created dynamically on the client-side using
     * the HTML5 &lt;canvas> element, and therefore requires a browser that supports
     * this element. It has been tested on IE9, Firefox 3.6/4 and
     * Chrome 10 browsers. If you can confirm whether it works on other browsers or
     * not, I'd love to hear from you!
     *
     * Usage:
     * The HeatMapLayer constructor requires:
     * - A reference to a map object
     * - An array or Microsoft.Maps.Location items
     * - Optional parameters to customise the appearance of the layer
     *  (Radius,, Unit, Intensity, and ColourGradient), and a callback function
     */
    
     var HeatMapLayer = function (map, locations, options) {
    
         /* Private Properties */
         var _map = map,
             _canvas,
             _temperaturemap,
             _locations = [],
             _viewchangestarthandler,
             _viewchangeendhandler;
    
         // Set default options
         var _options = {
             // Opacity at the centre of each heat point
             intensity: 0.5,
    
             // Affected radius of each heat point
             radius: 1000,
    
             // Whether the radius is an absolute pixel value or meters
             unit: 'meters',
    
             // Colour temperature gradient of the map
             colourgradient: {
                 "0.00": 'rgba(255,0,255,20)',  // Magenta
                 "0.25": 'rgba(0,0,255,40)',    // Blue
                 "0.50": 'rgba(0,255,0,80)',    // Green
                 "0.75": 'rgba(255,255,0,120)', // Yellow
                 "1.00": 'rgba(255,0,0,150)'    // Red
             },
    
             // Callback function to be fired after heatmap layer has been redrawn
             callback: null
         };
    
         /* Private Methods */
         function _init() {
             var _mapDiv = _map.getRootElement();
    
             if (_mapDiv.childNodes.length >= 3 && _mapDiv.childNodes[2].childNodes.length >= 2) {
                 // Create the canvas element
                 _canvas = document.createElement('canvas');
                 _canvas.style.position = 'relative';
    
                 var container = document.createElement('div');
                 container.style.position = 'absolute';
                 container.style.left = '0px';
                 container.style.top = '0px';
                 container.appendChild(_canvas);
    
                 _mapDiv.childNodes[2].childNodes[1].appendChild(container);
    
                 // Override defaults with any options passed in the constructor
                 _setOptions(options);
    
                 // Load array of location data
                 _setPoints(locations);
    
                 // Create a colour gradient from the suppied colourstops
                 _temperaturemap = _createColourGradient(_options.colourgradient);
    
                 // Wire up the event handler to redraw heatmap canvas
                 _viewchangestarthandler = Microsoft.Maps.Events.addHandler(_map, 'viewchangestart', _clearHeatMap);
                 _viewchangeendhandler = Microsoft.Maps.Events.addHandler(_map, 'viewchangeend', _createHeatMap);
    
                 _createHeatMap();
    
                 delete _init;
             } else {
                 setTimeout(_init, 100);
             }
         }
    
         // Resets the heat map
         function _clearHeatMap() {
             var ctx = _canvas.getContext("2d");
             ctx.clearRect(0, 0, _canvas.width, _canvas.height);
         }
    
         // Creates a colour gradient from supplied colour stops on initialisation
         function _createColourGradient(colourstops) {
             var ctx = document.createElement('canvas').getContext('2d');
             var grd = ctx.createLinearGradient(0, 0, 256, 0);
             for (var c in colourstops) {
                 grd.addColorStop(c, colourstops[c]);
             }
             ctx.fillStyle = grd;
             ctx.fillRect(0, 0, 256, 1);
             return ctx.getImageData(0, 0, 256, 1).data;
         }
    
         // Applies a colour gradient to the intensity map
         function _colouriseHeatMap() {
             var ctx = _canvas.getContext("2d");
             var dat = ctx.getImageData(0, 0, _canvas.width, _canvas.height);
             var pix = dat.data; // pix is a CanvasPixelArray containing height x width x 4 bytes of data (RGBA)
             for (var p = 0, len = pix.length; p < len;) {
                 var a = pix[p + 3] * 4; // get the alpha of this pixel
                 if (a != 0) { // If there is any data to plot
                     pix[p] = _temperaturemap[a]; // set the red value of the gradient that corresponds to this alpha
                     pix[p + 1] = _temperaturemap[a + 1]; //set the green value based on alpha
                     pix[p + 2] = _temperaturemap[a + 2]; //set the blue value based on alpha
                 }
                 p += 4; // Move on to the next pixel
             }
             ctx.putImageData(dat, 0, 0);
         }
    
         // Sets any options passed in
         function _setOptions(options) {
             for (attrname in options) {
                 _options[attrname] = options[attrname];
             }
         }
    
         // Sets the heatmap points from an array of Microsoft.Maps.Locations  
         function _setPoints(locations) {
             _locations = locations;
         }
    
         // Main method to draw the heatmap
         function _createHeatMap() {
             // Ensure the canvas matches the current dimensions of the map
             // This also has the effect of resetting the canvas
             _canvas.height = _map.getHeight();
             _canvas.width = _map.getWidth();
    
             _canvas.style.top = -_canvas.height / 2 + 'px';
             _canvas.style.left = -_canvas.width / 2 + 'px';
    
             // Calculate the pixel radius of each heatpoint at the current map zoom
             if (_options.unit == "pixels") {
                 radiusInPixel = _options.radius;
             } else {
                 radiusInPixel = _options.radius / _map.getMetersPerPixel();
             }
    
             var ctx = _canvas.getContext("2d");
    
             // Convert lat/long to pixel location
             var pixlocs = _map.tryLocationToPixel(_locations, Microsoft.Maps.PixelReference.control);
             var shadow = 'rgba(0, 0, 0, ' + _options.intensity + ')';
             var mapWidth = 256 * Math.pow(2, _map.getZoom());
    
             // Create the Intensity Map by looping through each location
             for (var i = 0, len = pixlocs.length; i < len; i++) {
                 var x = pixlocs[i].x;
                 var y = pixlocs[i].y;
    
                 if (x < 0) {
                     x += mapWidth * Math.ceil(Math.abs(x / mapWidth));
                 }
    
                 // Create radial gradient centred on this point
                 var grd = ctx.createRadialGradient(x, y, 0, x, y, radiusInPixel);
                 grd.addColorStop(0.0, shadow);
                 grd.addColorStop(1.0, 'transparent');
    
                 // Draw the heatpoint onto the canvas
                 ctx.fillStyle = grd;
                 ctx.fillRect(x - radiusInPixel, y - radiusInPixel, 2 * radiusInPixel, 2 * radiusInPixel);
             }
    
             // Apply the specified colour gradient to the intensity map
             _colouriseHeatMap();
    
             // Call the callback function, if specified
             if (_options.callback) {
                 _options.callback();
             }
         }
    
         /* Public Methods */
    
         this.Show = function () {
             if (_canvas) {
                 _canvas.style.display = '';
             }
         };
    
         this.Hide = function () {
             if (_canvas) {
                 _canvas.style.display = 'none';
             }
         };
    
         // Sets options for intensity, radius, colourgradient etc.
         this.SetOptions = function (options) {
             _setOptions(options);
         }
    
         // Sets an array of Microsoft.Maps.Locations from which the heatmap is created
         this.SetPoints = function (locations) {
             // Reset the existing heatmap layer
             _clearHeatMap();
             // Pass in the new set of locations
             _setPoints(locations);
             // Recreate the layer
             _createHeatMap();
         }
    
         // Removes the heatmap layer from the DOM
         this.Remove = function () {
             _canvas.parentNode.parentNode.removeChild(_canvas.parentNode);
    
             if (_viewchangestarthandler) { Microsoft.Maps.Events.removeHandler(_viewchangestarthandler); }
             if (_viewchangeendhandler) { Microsoft.Maps.Events.removeHandler(_viewchangeendhandler); }
    
             _locations = null;
             _temperaturemap = null;
             _canvas = null;
             _options = null;
             _viewchangestarthandler = null;
             _viewchangeendhandler = null;
         }
    
         // Call the initialisation routine
         _init();
     };
    
     // Call the Module Loaded method
     Microsoft.Maps.moduleLoaded('HeatMapModule');
    

To add twitterStream.js

  1. From Solution Explorer, expand TweetSentimentWeb.
  2. Right-click Scripts, click Add, click JavaScript File.
  3. In the Item name field, typetwitterStream.js.
  4. Copy and paste the following code into the file:

     var liveTweetsPos = [];
     var liveTweets = [];
     var liveTweetsNeg = [];
     var map;
     var heatmap;
     var heatmapNeg;
     var heatmapPos;
    
     function initialize() {
         // Initialize the map
         var options = {
             credentials: "AvFJTZPZv8l3gF8VC3Y7BPBd0r7LKo8dqKG02EAlqg9WAi0M7la6zSIT-HwkMQbx",
             center: new Microsoft.Maps.Location(23.0, 8.0),
             mapTypeId: Microsoft.Maps.MapTypeId.ordnanceSurvey,
             labelOverlay: Microsoft.Maps.LabelOverlay.hidden,
             zoom: 2.5
         };
         var map = new Microsoft.Maps.Map(document.getElementById('map_canvas'), options);
    
         // Heatmap options for positive, neutral and negative layers
    
         var heatmapOptions = {
             // Opacity at the centre of each heat point
             intensity: 0.5,
    
             // Affected radius of each heat point
             radius: 15,
    
             // Whether the radius is an absolute pixel value or meters
             unit: 'pixels'
         };
    
         var heatmapPosOptions = {
             // Opacity at the centre of each heat point
             intensity: 0.5,
    
             // Affected radius of each heat point
             radius: 15,
    
             // Whether the radius is an absolute pixel value or meters
             unit: 'pixels',
    
             colourgradient: {
                 0.0: 'rgba(0, 255, 255, 0)',
                 0.1: 'rgba(0, 255, 255, 1)',
                 0.2: 'rgba(0, 255, 191, 1)',
                 0.3: 'rgba(0, 255, 127, 1)',
                 0.4: 'rgba(0, 255, 63, 1)',
                 0.5: 'rgba(0, 127, 0, 1)',
                 0.7: 'rgba(0, 159, 0, 1)',
                 0.8: 'rgba(0, 191, 0, 1)',
                 0.9: 'rgba(0, 223, 0, 1)',
                 1.0: 'rgba(0, 255, 0, 1)'
             }
         };
    
         var heatmapNegOptions = {
             // Opacity at the centre of each heat point
             intensity: 0.5,
    
             // Affected radius of each heat point
             radius: 15,
    
             // Whether the radius is an absolute pixel value or meters
             unit: 'pixels',
    
             colourgradient: {
                 0.0: 'rgba(0, 255, 255, 0)',
                 0.1: 'rgba(0, 255, 255, 1)',
                 0.2: 'rgba(0, 191, 255, 1)',
                 0.3: 'rgba(0, 127, 255, 1)',
                 0.4: 'rgba(0, 63, 255, 1)',
                 0.5: 'rgba(0, 0, 127, 1)',
                 0.7: 'rgba(0, 0, 159, 1)',
                 0.8: 'rgba(0, 0, 191, 1)',
                 0.9: 'rgba(0, 0, 223, 1)',
                 1.0: 'rgba(0, 0, 255, 1)'
             }
         };
    
         // Register and load the Client Side HeatMap Module
         Microsoft.Maps.registerModule("HeatMapModule", "scripts/heatmap.js");
         Microsoft.Maps.loadModule("HeatMapModule", {
             callback: function () {
                 // Create heatmap layers for positive, neutral and negative tweets
                 heatmapPos = new HeatMapLayer(map, liveTweetsPos, heatmapPosOptions);
                 heatmap = new HeatMapLayer(map, liveTweets, heatmapOptions);
                 heatmapNeg = new HeatMapLayer(map, liveTweetsNeg, heatmapNegOptions);
             }
         });
    
         $("#searchbox").val("xbox");
         $("#searchBtn").click(onsearch);
         $("#positiveBtn").click(onPositiveBtn);
         $("#negativeBtn").click(onNegativeBtn);
         $("#neutralBtn").click(onNeutralBtn);
         $("#neutralBtn").button("toggle");
     }
    
     function onsearch() {
         var uri = 'api/tweets?query=';
         var query = $('#searchbox').val();
         $.getJSON(uri + query)
             .done(function (data) {
                 liveTweetsPos = [];
                 liveTweets = [];
                 liveTweetsNeg = [];
    
                 // On success, 'data' contains a list of tweets.
                 $.each(data, function (key, item) {
                     addTweet(item);
                 });
    
                 if (!$("#neutralBtn").hasClass('active')) {
                     $("#neutralBtn").button("toggle");
                 }
                 onNeutralBtn();
             })
             .fail(function (jqXHR, textStatus, err) {
                 $('#statustext').text('Error: ' + err);
             });
     }
    
     function addTweet(item) {
         //Add tweet to the heat map arrays.
         var tweetLocation = new Microsoft.Maps.Location(item.Latitude, item.Longtitude);
         if (item.Sentiment > 0) {
             liveTweetsPos.push(tweetLocation);
         } else if (item.Sentiment < 0) {
             liveTweetsNeg.push(tweetLocation);
         } else {
             liveTweets.push(tweetLocation);
         }
     }
    
     function onPositiveBtn() {
         if ($("#neutralBtn").hasClass('active')) {
             $("#neutralBtn").button("toggle");
         }
         if ($("#negativeBtn").hasClass('active')) {
             $("#negativeBtn").button("toggle");
         }
    
         heatmapPos.SetPoints(liveTweetsPos);
         heatmapPos.Show();
         heatmapNeg.Hide();
         heatmap.Hide();
    
         $('#statustext').text('Tweets: ' + liveTweetsPos.length + "   " + getPosNegRatio());
     }
    
     function onNeutralBtn() {
         if ($("#positiveBtn").hasClass('active')) {
             $("#positiveBtn").button("toggle");
         }
         if ($("#negativeBtn").hasClass('active')) {
             $("#negativeBtn").button("toggle");
         }
    
         heatmap.SetPoints(liveTweets);
         heatmap.Show();
         heatmapNeg.Hide();
         heatmapPos.Hide();
    
         $('#statustext').text('Tweets: ' + liveTweets.length + "   " + getPosNegRatio());
     }
    
     function onNegativeBtn() {
         if ($("#positiveBtn").hasClass('active')) {
             $("#positiveBtn").button("toggle");
         }
         if ($("#neutralBtn").hasClass('active')) {
             $("#neutralBtn").button("toggle");
         }
    
         heatmapNeg.SetPoints(liveTweetsNeg);
         heatmapNeg.Show();
         heatmap.Hide();;
         heatmapPos.Hide();;
    
         $('#statustext').text('Tweets: ' + liveTweetsNeg.length + "\t" + getPosNegRatio());
     }
    
     function getPosNegRatio() {
         if (liveTweetsNeg.length == 0) {
             return "";
         }
         else {
             var ratio = liveTweetsPos.length / liveTweetsNeg.length;
             var str = parseFloat(Math.round(ratio * 10) / 10).toFixed(1);
             return "Positive/Negative Ratio: " + str;
         }
     }
    

To modify the layout.cshtml

  1. From Solution Explorer, expand TweetSentimentWeb, expand Views, expand Shared, and then double-click _Layout.cshtml.
  2. Replace the content with the following:

     <!DOCTYPE html>
     <html>
     <head>
         <meta charset="utf-8" />
         <meta name="viewport" content="width=device-width, initial-scale=1.0">
         <title>@ViewBag.Title</title>
         @Styles.Render("~/Content/css")
         @Scripts.Render("~/bundles/modernizr")
         <!-- Bing Maps -->
         <script type="text/javascript" src="http://ecn.dev.virtualearth.net/mapcontrol/mapcontrol.ashx?v=7.0&mkt=en-gb"></script>
         <!-- Spatial Dashboard JavaScript -->
         <script src="~/Scripts/twitterStream.js" type="text/javascript"></script>
     </head>
     <body onload="initialize()">
         <div class="navbar navbar-inverse navbar-fixed-top">
             <div class="container">
                 <div class="navbar-header">
                     <button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-collapse">
                         <span class="icon-bar"></span>
                         <span class="icon-bar"></span>
                         <span class="icon-bar"></span>
                     </button>
                 </div>
                 <div class="navbar-collapse collapse">
                     <div class="row">
                         <ul class="nav navbar-nav col-lg-5">
                             <li class="col-lg-12">
                                 <div class="navbar-form">
                                     <input id="searchbox" type="search" class="form-control">
                                     <button type="button" id="searchBtn" class="btn btn-primary">Go</button>
                                 </div>
                             </li>
                         </ul>
                         <ul class="nav navbar-nav col-lg-7">
                             <li>
                                 <div class="navbar-form">
                                     <div class="btn-group" data-toggle="buttons-radio">
                                         <button type="button" id="positiveBtn" class="btn btn-primary">Positive</button>
                                         <button type="button" id="neutralBtn" class="btn btn-primary">Neutral</button>
                                         <button type="button" id="negativeBtn" class="btn btn-primary">Negative</button>
                                     </div>
                                 </div>
                             </li>
                             <li><span id="statustext" class="navbar-text"></span></li>
                         </ul>
                     </div>
                 </div>
             </div>
         </div>
         <div class="map_container">
             @RenderBody()
         </div>
         @Scripts.Render("~/bundles/jquery")
         @Scripts.Render("~/bundles/bootstrap")
         @RenderSection("scripts", required: false)
     </body>
     </html>
    

To modify the Index.cshtml

  1. From Solution Explorer, expand TweetSentimentWeb, expand Views, expand Home, and then double-click Index.cshtml.
  2. Replace the content with the following:

     @{
         ViewBag.Title = "Tweet Sentiment";
     }
    
     <div class="map_container">
         <div id="map_canvas"/>
     </div>
    

To modify the site.css file

  1. From Solution Explorer, expand TweetSentimentWeb, expand Content, and then double-click Site.css.
  2. Append the following code to the file:

     /* make container, and thus map, 100% width */
     .map_container {
         width: 100%;
         height: 100%;
     }
    
     #map_canvas{
       height:100%;
     }
    
     #tweets{
       position: absolute;
       top: 60px;
       left: 75px;
       z-index:1000;
       font-size: 30px;
     }
    

To modify the global.asax file

  1. From Solution Explorer, expand TweetSentimentWeb, and then double-click Global.asax.
  2. Add the following using statement:

     using System.Web.Http;
    
  3. Add the following lines inside the Application_Start() function:

     // Register API routes
     GlobalConfiguration.Configure(WebApiConfig.Register);
    

    Modify the registration of the API routes to make the Web API controller work inside the MVC application.

To run the web application

  1. Verify that the streaming service console application is still running so you can see the real-time changes.
  2. Press F5 to run the web application:

    hdinsight.hbase.twitter.sentiment.bing.map

  3. In the text box, enter a keyword, and then click Go. Depending on the data collected in the HBase table, some keywords might not be found. Try some common keywords, such as "love," "xbox," and "playstation."
  4. Toggle among Positive, Neutral, and Negative to compare sentiment on the subject.
  5. Let the streaming service run for another hour, and then search the same keywords, and compare the results.

Optionally, you can deploy the application to Azure Websites. For instructions, see Get started with Azure Websites and ASP.NET.

Next Steps

In this tutorial, you learned how to get tweets, analyze the sentiment of tweets, save the sentiment data to HBase, and present the real-time Twitter sentiment data to Bing maps. To learn more, see: