Lasttests - Lasttests leichtgemacht mit Microsoft Azure und Visual Studio Online

By Charles Sterling | November 2014

The Internet is the great equalizer in the business world. It helps the smallest companies compete with the largest corporations in the world. There is as much opportunity as there is risk. Applications must perform on time all the time. Downtime equates directly to lost revenue. To help development teams understand the performance characteristics of their apps under duress and heavy usage, Microsoft released load testing tools with Visual Studio Team System in 2005.

Even before that release, Visual Studio load testing was one of the most widely internally used test tools. Which begs the question, regardless of the tool used, why didn’t load testing gain the same acceptance by the world at large? Turns out load testing isn’t done by most development teams due to the effort of setting up and managing the infrastructure required. That’s an unnecessary risk.

With Visual Studio 2013, Microsoft released cloud-based load testing as part of Visual Studio Online. This solves the challenge of using Microsoft Azure to automatically provision and host a load testing infrastructure. With Visual Studio 2013 Update 3, cloud load testing in Visual Studio Online was further enhanced with Application Insights integration. This helps demonstrate internal diagnostics such as performance counters, exceptions and objects allocated during the run.

Create a Load Test

If you aren’t familiar with the load testing process in Visual Studio, you should start with a Web Performance and Load Test Project. Do this from the File | New Project menu. Select the Web Performance Test and Load Test to create one or more Web Tests.

To drive these Web Performance Tests, add a load test to the project (see Figure 1). Open Solution Finder, right-click on your project (mine is named WebLoadTestProject1), select Add, then Load Test and follow the directions in the New Load Test Wizard. During the load test, the system collects information such as how many users, duration, the type of network warm-up time, test distribution and so on. Another option is to create a dynamic load test using data binding (see “Data Binding Web Tests for Cloud-Based Load Testing”).

Figure 1 Add Load Test to a Project

Data Binding Web Tests for Cloud-Based Load Testing

While a static Web test can give you detailed performance information about your application, you’ll be able to derive more insight from a dynamic Web test. These dynamic load tests typically supply different values for each run. For example, instead of sending the search term “Frog” thousands of times, it would make more sense to use other likely search terms and names that test boundary conditions (such as “Toad,” “Frog” and so on). You can accomplish this with an advanced technique called data binding.

To data bind a Web test, you’ll need to add a data source to your Web test from the Web Test toolbar. This type of dynamic test will support XML, CSV and database as data sources. Keep in mind, for XML and CSV data sources the entire file is copied to the agent virtual machine. Also, for cloud-based load tests, the test agent needs to be able to connect to the data source. Therefore, the database data source needs to be running on Microsoft Azure SQL Database.

The properties of the data source you’re using for the Web test determine how the test will traverse the data source: Do Not Move, Random, Sequential or Unique. Note that the Unique property doesn’t yet work with cloud-based load tests. After you’ve added a data source, the value property of your query string parameter (previously the static string “Frog”) will now be a dropdown menu item. This will help you to bind the test to a particular column of your data source.

Put Azure to Work

Typically, the real work begins after you create the load test. At this point, the test team needs computers to host the Test Agents that actually run the load test. For example, if you want to put a 5,000-user load on your Web site and you can support 500 users per computer running the required tests, then your test team will need to acquire, provision and manage an additional 10 computers.

However, using the option in the Local.TestingSettings file, Visual Studio Online cloud load test can take care of all this for you. By default, cloud load test will automatically start an Azure-hosted agent for every 500 users. Each agent has two cores.

You can override this behavior through a new Run Settings property called Agent Count (Total Cores). The current implementation of cloud load test limits you to a maximum of 10 Agents or 20 cores. Cloud load test also requires at least 25 users per core. So while you can set the Agent Count (Total Cores) property to 10 for a load with only 10 users, cloud load test would only allocate one Agent and inform you of this with the following message:

“This load test will run using one agent cores. Agent Core Count value has been modified from ‘10’ to ‘one’ as a load test optimally needs ‘25’ virtual users per core. Learn more about it here at go.microsoft.com/fwlink/?LinkID=310093.”

For tests that incorporate more than 5,000 users, the users are distributed across the 10 agents and 20 cores.

Understand Your Load Test

You’ll typically start your load testing efforts by asking yourself questions the load test should answer like:

Is my application as fast as I expect?
Is my performance getting better or worse?
Is my code generating errors under load?

To answer these questions, Visual Studio load test provides several different reports. It will then automatically store them on the run data for both you and your team in Visual Studio Online.

For example, using the Throughput report and having that displayed during the load test execution, you’ll be able to tell at a glance if your application is running as expected (see Figure 2). Unexpectedly fast response times coupled with an unexpected number of failures would suggest a systemic failure (like a service being down or an incorrect server name). Then you can abort your load test.

Figure 2 The Visual Studio Online Load Test Report Page

Another challenging aspect of traditional load testing that Visual Studio load test solves is collaboration in a team environment, such as sharing and storing load test results. In the Load Test Manager in Figure 3, you can only see two team members are doing performance testing. The last four runs were aborted and they’ve just queued another run. Something isn’t going well there.

Figure 3 The Load Test Manager

Agile teams are typically interested in knowing the impact of their current sprint efforts on performance. To answer this, Visual Studio provides two different Excel-based reports that help the team compare runs and examine trends. To view this in-depth information, download the run data from Visual Studio Online. Click on the view report link. This opens a report view of your data—including the option to open and compare multiple sets of run data in Excel.

The report in Figure 4 compares Sprint2 versus Sprint3. You can see that while the performance is 4 percent slower, there are far fewer errors. That’s a manageable trade-off to the team.

Figure 4 An Example of a Completed Load Testing Report

Integrate Application Insights

Errors discovered during load testing are often questioned due to questionable diagnostics collection techniques. Worse yet are those errors that are difficult or impossible to diagnose due to insufficient diagnostics collection information.

With Visual Studio 2013 Update 2, cloud load testing was enhanced with the integration of Application Insights. See the May 2014 article, “Next-Generation Development with Application Insights” (msdn.microsoft.com/magazine/dn683794.aspx). This integration with production monitoring tools enables testing applications in development or production. It also helps development teams find and begin to address any errors they discover, regardless of when they were found—whether an hour, day or week earlier.

To include integrated performance and diagnostics information into your cloud load test using Application Insights, right-click on the Applications node of the load test definition. Then select the Application Insights Component you’d like to include in your load test reporting.

Wrapping Up

Cloud load testing in Visual Studio Online removes much of the work of setting up and load testing your application. While Azure makes it easy to scale applications and potentially “buy” your way out of poorly performing code, without load testing, it’s often difficult or impossible to know how much this will cost. It’s also difficult to tell if you’ll need to do this before there’s a negative impact on your customers. Full cloud load testing can help you identify live site bugs that can have an impact on your business.

Charles Sterling has been at Microsoft for 20 years. Before Microsoft he was a marine biologist working for the United States National Marine Fisheries doing marine mammal research on the Bering Sea. After a six-year tour in Australia as a Microsoft product manager, Sterling is currently back in Redmond as a senior program manager on the Visual Studio development team. Reach him at chass@microsoft.com.

Thanks to the following Microsoft technical expert for reviewing this article: Manas Maheshwari