Work with events from Social Engagement in Azure Event Hubs
Microsoft Social Engagement lets you stream posts to Azure Event Hubs and empowers you with data, so unleash your creativity! The options that Azure Event Hubs offer are huge. Benefit from a simple data format and highly performant cloud services to work with your data with endless possibilities. Build your own apps, connect your data with other data sources, and step into big data analysis.
To get you started, we’ve provided some inspiration to build a real-time Power BI Dashboard where we’ll stream posts from Microsoft Social Engagement to Azure Event Hubs. By using the power of Azure Stream Analytics we can consolidate the information and push it to Power BI to analyze our data in real time. Azure Stream Analytics provides the capabilities to combine the data from a variety of sources, where Azure Event Hubs is one of them. It can send the data to another event hub, a SQL database, or to Power BI. Power BI lets you build your own business intelligence based on the data sources you are connected to. All connections listed in the scenario below are supported by default with the services. No need to customize—get started right away!
For the following scenario, we assume that you have access to Power BI and an Azure subscription (including Azure Service Bus, Azure Event Hubs, and Azure Stream Analytics), and have admin permissions to work through the steps. If you want to know more about the functionality of these services, please refer to the following links:
Analyze social posts in real time with Power BI
Learn more about the power and flexibility of the integration with Azure Event Hubs, connecting Microsoft services across the board. In this lightweight example, we’ll connect a data set of social posts to Power BI to perform real-time analysis. Make sure that you have access to the required services with the right privileges (usually admin permissions).
|Create the event hub you will stream data to.
You can skip this step if you already have an event hub to work with.
|Step 1: Create an event hub|
|Establish the connection between Microsoft Social Engagement and Azure Event Hubs.||Step 2: Connect Social Engagement to the event hub|
|Define the data set that gets streamed as events to Azure Event Hubs.||Step 3: Create an automation rule to select the posts that get streamed to the event hub|
|Create an Azure Stream Analytics job and write the query to further process your data.||Step 4: Create an Azure Stream Analytics job|
|Send the data to a Power BI dashboard.||Step 5: Create a dashboard in Power BI|
Step 1: Create an event hub
Create a new namespace in Service Bus. Select Type: Messaging, and then choose the other values to your preference. You can also use an existing Service Bus namespace.
Create a new event hub in the namespace.
To get the connection string, click Connection Information in the Event Hub view in Azure Portal. If it’s a new setup, you need to configure the Shared Access Signature (SAS) Policies first.
Create a new SAS policy under shared access policies. Give it a name, add a rule named SendRule with Send rights, and then click Save.
Copy the Connection String for SendRule; we’ll need it again in the next step.
Step 2: Connect Social Engagement to the event hub
Now that the event hub is ready to receive data, you need to connect Social Engagement to your event hub using the Connection String provided for your event hub in the Azure Portal. More information: Stream data from Social Engagement to Microsoft Azure Event Hubs
Step 3: Create an automation rule to select the posts that get streamed to the event hub
In Social Engagement, create an automation rule that streams events to your event hub. Make sure the filters are defined according to your requirements so you get the posts that you are looking for. More information: Route posts using automation rules, Get relevant data using filters
In Social Engagement, go to Settings > Automation Rules, and then create an automation rule that streams events to your event hub.
In Azure Portal, check the dashboard of the event hub and verify that the events from Social Engagement appear. Depending on the selected data set, it may take a while for new posts to get published on social media and pushed to the event hub.
Step 4: Create an Azure Stream Analytics job
Let's go ahead and connect the event hub and Azure Stream Analytics.
In the Azure Portal, create a new Azure Stream Analytics job.
Name the job, and then choose the Region and the Regional Monitoring Storage Account.
Go to Stream Analytics Job details and define the input source as Data Stream > Event Hub. Provide an alias for your input and choose the namespace created in Step 1.
For Event Hub Policy name, select RootManagedSharedAccessKey. Make a note of your input alias; you’ll need it later.
Keep the Serialization settings to JSON and UTF8.
Add an output sink for Stream Analytics. For our scenario, select Power BI. Provide an alias for your output and make a note of it; we’ll need it later. When you select Power BI, you need to authorize your account, or create a new subscription.
Sign in to the Office 365 portal and open Power BI first to establish a fresh session with Power BI before you choose it as the output sink.
Now for the tricky part: We need to tell the Stream Analytics job what data it should look for and what it should pass on to Power BI. This is done with a SQL statement, which selects the events that match certain conditions (the selectors). The resulting data gets pushed to the connected service (in this case Power BI), to store the events generated by Stream Analytics. Refer to our examples below or develop your own Stream Analytics SQL statement to query the data on the event hub and aggregate the data set you want to push to the output sink. More information: MSDN: Stream Analytics Query Language Reference
Windows Dev Center hosts the Service Bus Explorer—a tool to manage Azure Service Bus components. More information: Service Bus Explorer Download
Connect with Service Bus Explorer to your Azure Service Bus namespace.
In the Azure Portal, go to Service Bus > [namespace] > Connection Information. Copy the connection string from your SAS key for RootManagedSharedAccessKey to connect to your namespace.
In Service Bus Explorer, expand the event hub you connected to Social Engagement. Right-click the $Default consumer group, and then click Create Consumer Group Listener. Start the listener, and then review the JSON payload in the Event Text field of the Events tab.
More information: JSON reference for events from Social Engagement
Example 1: In this simple “Hello World” example, we count all new posts over time and push them every 30 seconds to the output sink.
SELECT count (*) AS count, System.TimeStamp AS Time INTO [your-output-sink-alias] FROM [your-stream-input-alias] GROUP BY TumblingWindow(second,30)
Example 2: In this example, we select unique publishers and their country/region information. The results will get pushed to the output sink every 30 seconds.
WITH current_window AS ( SELECT post.profile.profileLocation.adminDistrict, post.profile.profileLocation.countryRegion, count (*) AS count FROM [your-stream-input-alias] GROUP BY post.profile.profileLocation.adminDistrict, post.profile.profileLocation.countryRegion, TumblingWindow (second,30) ) SELECT *, System.Timestamp AS Time INTO [your-output-sink-alias] FROM current_window
Example 3: In this example, we select the name of a matching search topic in addition to the id, title, acquisition date and time, source name, and the content excerpt of posts. We're using the GetArrayElement function to select the first object of the post.matchingSearchTopic array. To select all objects of the array, you can use the GetArrayElements function.
WITH sub_query AS ( SELECT post.id, post.title, post.acquisitionDate, post.source.param, post.sentiment.polarity, System.TimeStamp AS time, GetArrayElement(post.matchingSearchTopics, 0) AS SearchTopic, post.content.text FROM [your-stream-input-alias] ) SELECT id, title, acquisitionDate, param AS source, polarity, time, SearchTopic.name AS topic, text INTO [your-output-sink-alias] FROM sub_query
Step 5: Create a dashboard in Power BI
Let’s close the loop. Since our goal is to create a real-time dashboard in Power BI, we need to make sure it receives data from Stream Analytics. When we defined the output sink, we authorized Power BI with the Stream Analytics job. A data set in Power BI was already created. Go to Power BI and select the data set you defined in step 4.5. The name of the data set matches the name of your Stream Analytics output sink.
In Power BI, create a new visualization.
For the example 1, in step 4, it’s a line chart.
For the example 2, in step 4, we can select a map to visualize the data.
Location: countryregion or admindisctrict
There are almost unlimited options to combine data from Social Engagement with data from other applications or the Internet. The following picture shows you a Power BI dashboard that correlates weather data with the different types of intentions from Social Engagement's machine-learning based intention analysis.
By enabling Social Engagement to connect to Azure Event Hubs, you will allow social data to be streamed to event hubs using automation rules. Azure Event Hubs stores the social data streamed from Social Engagement for a pre-configured period of time and any application that can listen to the event hub will be able to access, store and/or process this data.
Note that the social data sent from Social Engagement includes information regarding the social post (author and text) as well as enriched information such as Language, Location, Sentiment, Tags, etc. For complete information on the content of a social post sent to event hubs, please see the JSON schema definition.