Changes to load test functionality in Visual Studio and cloud load testing in Azure DevOps

Azure Test Plans

Web performance and load test functionality in Visual Studio is deprecated. Visual Studio 2019 will be the last version of Visual Studio containing web performance and load test features (more details).

Cloud load testing (CLT) functionality in Azure DevOps is deprecated. The changes to cloud load testing functionality in Azure DevOps include all entry points for the load testing service:

  1. Web-based experience for load testing in Azure DevOps (URL-based, HAR file, Apache JMeter and Visual Studio web test).

  2. Running a load test in the cloud using Visual Studio (both auto-provisioned agents and self-provisioned agents).

  3. Running performance tests using the Azure portal (performance test in App Services and Application Insights).

  4. Running load tests in a CI/CD pipeline using the load testing tasks (Cloud-based Load Test task, Apache JMeter Test task, and Cloud-based Web Performance Test task).

Timeframe

On-premises load testing with Visual Studio and Test Controller/Test Agent

Visual Studio 2019 will be the last version of Visual Studio with the web performance and load test capability. Visual Studio 2019 is also the last release for Test Controller and Test Agent (installed through the 'Agents for Visual Studio SKU') for setting up a load test rig on-premises.

While no new features will be added, load test in VS 2019 will continue to be supported for any issues that may arise during the support lifecycle of the product. See Product Lifecycle and Servicing.

Cloud-based load testing service (CLT) availability timeframe for:

  1. Existing customers of CLT:

    • For existing Azure DevOps customers using cloud-load testing functionality in the Azure DevOps portal, this feature will no longer be available as of March 31, 2020.

    • For existing Azure customers using the performance test feature in App Services or Application Insights, this feature will no longer be available as of March 31, 2020.

    • For Visual Studio users leveraging cloud-based load testing, this feature will no longer be supported as of March 31, 2020. Visual Studio Enterprise customers can, however, continue to run load tests on-premises using Test Controller/Test Agent and will be supported for any issues that may arise during the support lifecycle of the Visual Studio version.

  2. New customers of CLT:

    New Azure DevOps organizations and new Azure subscriptions created after March 31st, 2019 will not have access to the cloud-based load testing functionality.

Alternatives

For customers who prefer to continue using a self-service model for load test and cloud-load testing, there are a number of free and commercial tools available.

For example, Apache JMeter is a free, popular open source tool with a strong community backing. It supports many different protocols and has rich extensibility that can be leveraged to customize the tool to your needs. In addition, if you have been using Apache JMeter to run load tests using the cloud-based load testing service, you will be able to use these load tests with alternatives that support them.

The following table outlines how the concepts in Visual Studio load test map to those in Apache JMeter.

Visual Studio Load test Apache JMeter load test
Load test (.loadtest file) JMX (.jmx file)
Webtest recorder HTTP(S) test script recorder
Webtest HTTP sampler
Unit test JUnit sampler
Selenium test WebDriver sampler
Request and test plugins for extensibility Extensibility using JMeter plugins
Reporting Dashboard
Distribution using Test Controller/Test Agent Distribution in Apache JMeter

Many commercial services such as Blazemeter support running Apache JMeter tests.

If you use code based tests for load testing and .NET is your platform of choice then tools such as Neoload, Micro Focus Silk Performer, and Micro Focus Load Runner are options to consider.

In addition, extensions from several load test vendors such as SOASTA (now Akamai CloudTest), Apica Loadtest, and Load Impact are available in the Azure DevOps and Azure marketplace.

Download load tests and results

If you are using URL, HAR, or Visual Studio webtest-based load tests, you can download the tests by using the Export test to Visual Studio option and use the test through Visual Studio during the timeframe outlined above.

If you are using the cloud-based load testing service and want to access to results of any existing runs after the service goes offline, you can download the reports:

  • If you use Visual Studio load tests, URL-based load tests, or HAR-based load tests, download the reports in Visual Studio Enterprise for offline viewing.

  • If you use Apache JMeter, download the results file (.csv) for offline viewing using the reporters in JMeter.

Visual Studio web performance test (.webtest file)

Visual Studio web performance test (.webtest file) is tied to the load test functionality and is deprecated. Some customers have used .webtest for other purposes such as running API tests, even though it was not designed for that purpose. Many API testing alternatives are available in the market. SOAP UI is a free, open source alternative to consider, and is also available as a commercial option with additional capabilities.