In this tutorial, you learn how to use Azure Stream Analytics Tools for Visual Studio to create, author, test locally, manage, and debug your Stream Analytics jobs.
After completing this tutorial, you will be able to:
- Familiarize yourself with Stream Analytics Tools for Visual Studio.
- Configure and deploy a Stream Analytics job.
- Test your job locally with local sample data.
- Use monitoring to troubleshoot issues.
- Export existing jobs to projects.
To complete this tutorial, you need the following prerequisites:
- Finish the steps that precede "Create a Stream Analytics job" in the Build an IoT solution by using Stream Analytics tutorial.
- Use Visual Studio 2015, Visual Studio 2013 update 4, or Visual Studio 2012. Enterprise (Ultimate/Premium), Professional, and Community editions are supported. Express edition is not supported. Visual Studio 2017 is not supported.
- Use the Azure SDK for .NET version 2.7.1 or later. Install it by using the Web platform installer.
- Install the Stream Analytics Tools for Visual Studio.
Create a Stream Analytics project
In Visual Studio, click the File menu and select New Project.
In the templates list on the left, select Stream Analytics and then click Azure Stream Analytics Application.
Enter the project Name, Location, and Solution name as you do for other projects.
A Toll project is generated in Solution Explorer.
Choose the correct subscription
In Visual Studio, click the View menu and open Server Explorer.
Sign in with your Azure Account.
Define the input sources
- In Solution Explorer, expand the Inputs node and rename Input.json to EntryStream.json. Double-click EntryStream.json.
- The Input Alias is now EntryStream. The input alias is used in the query script.
- In Source Type, select Data Stream.
- In Source, select Event Hub.
- In Service Bus Namespace, select the TollData option.
- In Event Hub Name, select entry.
- In Event Hub Policy Name, select RootManageSharedAccessKey (the default value).
- In Event Serialization Format, select Json.
In Encoding, select UTF8. Your settings should look like the following screenshot:
To finish the wizard, click Save. Now you can add another input source to create the exit stream. Right-click the Inputs node, and select New Item.
In the window, select Azure Stream Analytics Input, and change the Name to ExitStream.json. Click Add.
Double-click ExitStream.json in the project, and follow the same steps as you did for the entry stream. Be sure to enter exit for the Event Hub Name as shown in the following screenshot:
Now you have defined two input streams:
Next, add reference data input for the blob file that contains car registration data.
Right-click the Inputs node in the project, and then follow the same steps as you did for the stream inputs. In Input Alias, enter Registration, and in Source Type, select Reference data.
In Storage Account, select the tolldata option. In Container, select tolldata, and in Path Pattern, enter registration.json. This file name is case sensitive and should be lowercase.
- To finish the wizard, click Save.
Now all the inputs are defined.
Define the output
In Solution Explorer, expand the Inputs node and double-click Output.json.
In Output Alias, enter output.
- In Sink, select SQL Database.
- In Database, select TollDataDB.
- In User Name, enter tolladmin.
- In Password, enter 123toll!.
- In Table, enter TollDataRefJoin.
Create a Stream Analytics query
This tutorial attempts to answer several business questions that are related to toll data. It also constructs Stream Analytics queries that can be used in Stream Analytics to provide relevant answers. Before you start your first Stream Analytics job, let’s explore a simple scenario and the query syntax.
Introduction to the Stream Analytics query language
Let’s say that you need to count the number of vehicles that enter a toll booth. Because this example is a continuous stream of events, you have to define a period of time. Modify the question to be “How many vehicles enter a toll booth every three minutes?” This way to count data is commonly referred to as the tumbling count.
Look at the Stream Analytics query that answers this question:
SELECT TollId, System.Timestamp AS WindowEnd, COUNT(*) AS Count FROM EntryStream TIMESTAMP BY EntryTime GROUP BY TUMBLINGWINDOW(minute, 3), TollId
Stream Analytics uses a query language that's like SQL and adds a few extensions to specify time-related aspects of the query.
Now that you have written your first Stream Analytics query, it's time to test it. Use the sample data files located in your TollApp folder in the following path:
This folder contains the following files:
Count the number of vehicles entering a toll booth
In the project, double-click Script.asaql to open the script in the Query Editor. Copy and paste the script in the previous section into the editor. The Query Editor supports IntelliSense, syntax coloring, and the error marker.
Test Stream Analytics queries locally
To compile the query to see if there is a syntax error, right-click the project and select Build.
To validate this query against sample data, you can use local sample data. Right-click the input, and select Add local input.
In the pop-up window, select the sample data from your local path. Click Save.
A file named local_EntryStream.json is automatically added to your inputs folder.
In the Query Editor, click Run Locally. Or you can press the F5 key.
Press any key to view the output in the ASA Local Run Result window in Visual Studio.
Click Open Result Folder to check the output files both in CSV and JSON format.
Sample the input data
You can also sample input data from input sources to a local file.
Right-click the input config file, and select Sample Data.
You can sample only event hub or IoT hub for now. Other input sources are not supported.
In the pop-up window, enter the local path used to save the sample data. Click Sample.
You can see the progress in the Output window.
Submit a Stream Analytics query to Azure
In the Query Editor, click Submit To Azure in the script editor.
Select Create a New Azure Stream Analytics Job. Enter the Job Name, and select the correct Subscription. Click Submit.
Start a job
Now that your job is created, the job view is automatically opened.
To start the job, click the green arrow button.
Select the default setting, and click Start.
The job Status changes to Running, and Input Events and Output Events appear.
Check the results in Visual Studio
- In Visual Studio, open Server Explorer and right-click the TollDataRefJoin table.
Select Show Table Data to see the output of your job.
View the job metrics
Some basic job statistics can be found in Job Metrics.
List the job in Server Explorer
In Server Explorer, click Stream Analytics Jobs and then click Refresh. The job appears under Stream Analytics jobs.
Open the job view
To open the job view, expand your job node and double-click the Job View node.
Export an existing job to a project
There are two ways you can export an existing job to a project.
In Server Explorer, right-click the job node in the Stream Analytics Jobs node and select Export to New Stream Analytics Project.
The project is generated in Solution Explorer.
You also can use the job view, and click Generate Project.
Known issues and limitations
- There is no support for Power BI output and Azure Date Lake Store output.