Tutorial: Identify performance regressions with Azure Pipelines and Azure Load Testing Preview

Article
11/30/2021
11 minutes to read

In this tutorial, you'll learn how to automate performance regression testing with Azure Load Testing Preview and Azure Pipelines. You'll configure an Azure Pipelines CI/CD workflow to run a load test for a sample web application, and then use the results to identify performance regressions.

If you're using GitHub Actions for your CI/CD workflows, see the corresponding GitHub Actions tutorial.

You'll learn how to:

Set up your repository with files required for load testing.
Set up Azure Pipelines to integrate with Azure Load Testing.
Run the load test and view results in the pipeline logs.
Define pass/fail criteria for the load test.
Parameterize the load test using pipeline variables.

Important

Azure Load Testing is currently in PREVIEW. See the Supplemental Terms of Use for Microsoft Azure Previews for legal terms that apply to Azure features that are in beta, preview, or otherwise not yet released into general availability.

Note

Azure Pipelines has a 60-minute timeout on jobs that are running on Microsoft-hosted agents for private projects. If your load test is running for more than 60 minutes, you'll need to pay for additional capacity. If not, the pipeline will time out without waiting for the test results. You can view the load test status in the Azure portal.

Prerequisites

An Azure account with an active subscription. If you don't have an Azure subscription, create a free account before you begin.
An Azure DevOps organization and project. If you don't have an Azure DevOps organization, you can create one for free. If you need help with getting started with Azure Pipelines, see Create your first pipeline.
A GitHub account, where you can create a repository. If you don't have one, you can create one for free.
An existing Azure Load Testing resource. To create a Load Testing resource, see Create and run a load test.

Set up your repository

To get started, you first need a GitHub repository with the sample web application. You'll then use this repository to configure an Azure Pipelines workflow to run the load test.

Open a browser and navigate to the sample application's source GitHub repository: https://github.com/Azure-Samples/nodejs-appsvc-cosmosdb-bottleneck.git

The sample application is a Node.js app consisting of an Azure App Service web component and a Cosmos DB database.
Select Fork to fork the sample application repository to your GitHub account.

Configure the Apache JMeter script

The sample application's source repo includes an Apache JMeter script named SampleApp.jmx. This script carries out three API calls on each test iteration:

add - Carries out a data insert operation on Cosmos DB for the number of visitors on the webapp.
get - Carries out a GET operation from Cosmos DB to retrieve the count.
lasttimestamp - Updates the timestamp since the last user went to the website.

In this section, you'll update the Apache JMeter script with the URL of your sample web app.

In your sample application repository, open SampleApp.jmx for editing.
Search for <stringProp name="HTTPSampler.domain">.

You'll see three instances of <stringProp name="HTTPSampler.domain"> in the file.
Replace the value with the URL of your sample application's App Service in all three instances.
```
<stringProp name="HTTPSampler.domain">{your-app-name}.azurewebsites.net</stringProp>
```
You'll deploy the sample application to an Azure App Service web app by using Azure Pipelines in the subsequent steps. For now, replace the placeholder text {your-app-name} in the previous previous XML snippet with a unique name you wish to provide to the App Service web app. You'll then use this same name to create the web app.

Important

Don't include https or http in the sample application's URL.
Commit your changes to the main branch.

Set up Azure Pipelines access permissions for Azure

In this section, you'll configure your Azure DevOps project to have permissions to access the Azure Load Testing resource.

To access Azure resources, you'll create a service connection in Azure DevOps and use role-based access control to assign the necessary permissions.

Sign in to your Azure DevOps organization (https://dev.azure.com/<yourorganization>).
Select Project settings > Service connections.
Select + New service connection, select the Azure Resource Manager service connection, and then select Next.
Select the Service Principal (automatic) authentication method, and then select Next.
Select the Subscription scope level and then select your Azure subscription that contains your Azure Load Testing resource.

You'll use the name of the service connection in a later step to configure the pipeline.
Select Save to create the connection.
Select the service connection from the list, and then select Manage Service Principal.

You'll see the details of the service principal in the Azure portal. Note the service principal Application (Client) ID.
Now, assign the Load Test Contributor role to the service principal to allow access to the Azure Load Testing service.

First, retrieve the service principal object ID. Select the objectId result from the following Azure CLI command:
```
az ad sp show --id "<application-client-id>"
```
Replace the placeholder text <sp-object-id> with the service principal object ID. Also, replace the <subscription-name-or-id> with your Azure subscription ID.
```
az role assignment create --assignee "<sp-object-id>" \
    --role "Load Test Contributor" \
    --subscription "<subscription-name-or-id>"
```

Configure the Azure Pipelines workflow to run a load test

In this section, you'll set up an Azure Pipelines workflow that triggers the load test.

First, you'll install the Azure Load Testing extension from the Azure DevOps Marketplace, create a new pipeline, and then connect it to the sample application's forked repository.

Install the Azure Load Testing task extension from the Azure DevOps Marketplace.
In your Azure DevOps project, select Pipelines, and then select Create pipeline.
In the Connect tab, select GitHub.
Select Authorize AzurePipelines to allow Azure Pipelines to access your GitHub account for triggering workflows.
In the Select tab, select the sample application's forked repository.

The repository contains an azure-pipeline.yml pipeline definition file. You'll now modify this definition to connect to your Azure Load Testing service.

In the Review tab, replace the following placeholder texts in the YAML code.

Placeholder	Value
`<Name of your webapp>`	The Azure App Service web app name. This name should match the name used for the endpoint URL in the SampleApp.jmx test script.
`<Name of your webARM Service connection>`	The name of the service connection you created in the previous section.
`<Azure subscriptionId>`	Your Azure subscription ID.
`<Name of your load test resource>`	The name of your Azure Load Testing resource.
`<Name of your load test resource group>`	The resource group name that contains the Azure Load Testing resource.

Important

The name of Azure web app should match the name you used for the endpoint URL in the SampleApp.jmx test script.

Screenshot that shows the Azure Pipelines Review tab when creating a pipeline.

Select Save and run, enter a Commit message, and then select Save and run.

Azure Pipelines now runs the CI/CD workflow. You can monitor the status and logs by selecting the pipeline job.

View load test results

For every update to the main branch, the Azure pipeline executes the following steps:

Deploy the sample Node.js application to an Azure App Services web app. The name of the web app is configured in the pipeline definition.
Trigger Azure Load Testing to create and run the load test, based on the Apache JMeter script and the test configuration YAML file in the repository.

In this section, you'll view the load test results in the pipeline log information.

In your Azure DevOps project, select Pipelines, and then select your pipeline definition from the list.
Select the pipeline run to view the run summary.
Select Load Test in the Jobs section to view the pipeline logs.

Once the load test finishes, you can view the test summary information and the client-side metrics in the pipeline logs. The log also shows the URL to navigate to the Azure Load Testing dashboard for this load test.
In the pipeline log view, select Load Test, and then select 1 artifact produced to download the load test result files.

Define test pass/fail criteria

In this section, you'll add failure criteria to determine the outcome of your load test. If at least one of the failure criteria evaluates to true, the load test is unsuccessful.

You can specify these criteria in the test configuration YAML file.

Edit the SampleApp.yml file in your GitHub repository.
Add the following snippet at the end of the file.
```
failureCriteria: 
    - avg(response_time_ms) > 100
    - percentage(error) > 20
```
You've now specified failure criteria for your load test. The test will fail if at least one of these conditions is met:
- The aggregate average response time is greater than 100 ms.
- The aggregate percentage of errors is greater than 20%.
Commit and push the changes to the main branch of the repository.

The changes will trigger the Azure Pipelines CI/CD workflow.
In the pipeline runs page, select the most recent entry from the list.

After the load test finishes, you'll notice that the pipeline failed because the average response time was higher than you specified in the failure criteria.

The Azure Load Testing service evaluates the criteria during the test execution. If any of these conditions fails, Azure Load Testing service returns a non-zero exit code. This code informs the CI/CD workflow that the test has failed.

Edit the SampleApp.yml file and change the test failure criteria.

failureCriteria: 
    - avg(response_time_ms) > 5000
    - percentage(error) > 20

Commit the changes to trigger the Azure Pipelines CI/CD workflow.

The load test now succeeds and the pipeline finishes successfully.

Pass parameters to your load tests from the pipeline

Next, you'll parameterize your load test using pipeline variables. These variables can be secrets, such as passwords, or non-secrets.

In this tutorial, you'll reconfigure the sample application to only accept secure requests. To send a secure request, you'll need to pass a secret value in the HTTP request.

Edit the SampleApp.yaml file in your GitHub repository.

Update the testPlan configuration setting to use the SampleApp_Secrets.jmx file.
```
version: v0.1
testName: SampleApp
testPlan: SampleApp_Secrets.jmx
description: 'SampleApp Test with secrets'
engineInstances: 1
```
The SampleApp_Secrets.jmx Apache JMeter script uses a user-defined variable that retrieves the secret value with the custom function ${__GetSecret(secretName)}. Apache JMeter then passes this secret value to the sample application endpoint.
Commit the changes to the YAML file.
Edit the config.json file in your GitHub repository.

Update the enableSecretsFeature setting to true to reconfigure the sample application to only accept secure requests.
```
{
    "enableSecretsFeature": true
}
```
Commit the changes to the config.json file.
Edit the SampleApp_Secrets.jmx file.
Search for <stringProp name="HTTPSampler.domain">.

You'll see three instances of <stringProp name="HTTPSampler.domain"> in the file.
Replace the value with the URL of your sample application's App Service in all three instances.
```
<stringProp name="HTTPSampler.domain">{your-app-name}.azurewebsites.net</stringProp>
```
You'll deploy the sample application to an Azure App Service web app by using the GitHub Actions workflow. In the previous XML snippet, replace the placeholder text {your-app-name} with the unique name of the App Service web app.

Important

Don't include https or http in the sample application's URL.
Save and commit the Apache JMeter script.
Go to the Pipelines page, select your pipeline definition, and then select Edit.
Select Variables, and then select New variable.
Enter the Name (mySecret) and Value (1797669089) information, and then check the Keep this value secret box to store the variable securely.
Select OK, and then select Save to save the new variable.
Edit the azure-pipeline.yml file to pass the secret to the load test.

Edit the Azure Load Testing task by adding the following YAML snippet:
```
secrets: |
  [
      {
      "name": "appToken",
      "value": "$(mySecret)"
      }
  ]
```
Save and run the pipeline.

The Azure Load Testing task securely passes the secret from the pipeline to the test engine. The secret parameter is only used while running the load test, and then the value is discarded from memory.

Take a look at the overview of the Azure Load Testing task in the next section.

Azure Load Testing Task

This section describes the Azure Load Testing task for Azure Pipelines. The task is cross-platform, and runs on Windows, Linux, or Mac agents.

You can use the following parameters to configure the Azure Load Testing task.

Parameter	Description
`azureSubscription`	Required Name of the Azure Resource Manager service connection.
`loadTestConfigFile`	Required Path to the load test YAML configuration file. The path is fully qualified or relative to the default working directory.
`resourceGroup`	Required Name of the resource group that contains the Azure Load Testing resource.
`loadTestResource`	Required Name of an existing Azure Load Testing resource.
`secrets`	Array of JSON objects that consist of the name and value for each secret. The name should match the secret name used in the Apache JMeter test script.
`env`	Array of JSON objects that consist of the name and value for each environment variable. The name should match the variable name used in the Apache JMeter test script.

The following YAML code snippet describes how to use the task in an Azure Pipelines CI/CD workflow.

- task: AzureLoadTest@1
  inputs:
    azureSubscription: '<Azure service connection>'
    loadTestConfigFile: '< YAML File path>'
    loadTestResource: '<name of the load test resource>'
    resourceGroup: '<name of the resource group of your load test resource>' 
    secrets: |
      [
          {
          "name": "<Name of the secret>",
          "value": "$(mySecret1)",
          },
          {
          "name": "<Name of the secret>",
          "value": "$(mySecret1)",
          }
      ]
    env: |
      [
          {
          "name": "<Name of the variable>",
          "value": "<Value of the variable>",
          },
          {
          "name": "<Name of the variable>",
          "value": "<Value of the variable>",
          }
      ]

Clean up resources

Important

You can reuse the Azure Load Testing resource that you created for other Azure Load Testing tutorials and how-to articles.

If you don't plan to use any of the resources that you created, delete them so you don't incur any further charges. If you've deployed the sample application in a different resource group, you might want to repeat the following steps.

In the Azure portal:
1. Select the menu button in the upper-left corner, then select Resource groups.
2. From the list, select the resource group that you created.
3. Select Delete resource group.
4. Enter the resource group name. Then select Delete.
Alternately, you can use the Azure CLI.
```
az group delete --name <yourresourcegroup>
```
Remember, deleting the resource group deletes all of the resources within it.

Next steps

You've now created an Azure Pipelines CI/CD workflow that uses Azure Load Testing for automatically running load tests. By using pass/fail criteria, you can set the status of the CI/CD workflow. With parameters, you can make the load test execution configurable.

For more information about parameterizing load tests, see Parameterize a load test.
For more information about defining test pass/fail criteria, see Define test criteria.

Tutorial: Identify performance regressions with Azure Pipelines and Azure Load Testing Preview

Please rate your experience

Prerequisites

Set up your repository

Configure the Apache JMeter script

Set up Azure Pipelines access permissions for Azure

Configure the Azure Pipelines workflow to run a load test

View load test results

Define test pass/fail criteria

Pass parameters to your load tests from the pipeline

Azure Load Testing Task

Clean up resources

Next steps

Feedback