Tutorial: Identify performance regressions with Azure Load Testing Preview and Azure Pipelines

This tutorial describes how to automate performance regression testing by using Azure Load Testing Preview and Azure Pipelines. You'll configure an Azure Pipelines continuous integration and continuous delivery (CI/CD) workflow to run a load test for a sample web application. You'll then use the test results to identify performance regressions.

If you're using GitHub Actions for your CI/CD workflows, see the corresponding GitHub Actions tutorial.

You'll learn how to:

Prerequisites

• An Azure account with an active subscription. If you don't have an Azure subscription, create a free account before you begin.
• An Azure DevOps organization and project. If you don't have an Azure DevOps organization, you can create one for free. If you need help with getting started with Azure Pipelines, see Create your first pipeline.
• A GitHub account, where you can create a repository. If you don't have one, you can create one for free.
• An existing Azure Load Testing resource. To create a Load Testing resource, see Create and run a load test.

Set up your repository

To get started, you need a GitHub repository with the sample web application. You'll use this repository to configure an Azure Pipelines workflow to run the load test.

1. Open a browser and go to the sample application's source GitHub repository.

   The sample application is a Node.js app that consists of an Azure App Service web component and an Azure Cosmos DB database.

2. Select Fork to fork the sample application's repository to your GitHub account.

   

Configure the Apache JMeter script

The sample application's source repo includes an Apache JMeter script named SampleApp.jmx. This script makes three API calls on each test iteration:

• add: Carries out a data insert operation on Azure Cosmos DB for the number of visitors on the web app.
• get: Carries out a GET operation from Azure Cosmos DB to retrieve the count.
• lasttimestamp: Updates the time stamp since the last user went to the website.

Update the Apache JMeter script with the URL of your sample web app:

1. In your sample application's repository, open SampleApp.jmx for editing.

   

2. Search for <stringProp name="HTTPSampler.domain">.

   You'll see three instances of <stringProp name="HTTPSampler.domain"> in the file.

3. Replace all three instances of the value with the URL of your sample web app:

   <stringProp name="HTTPSampler.domain">{your-app-name}.azurewebsites.net</stringProp>
   

   You'll deploy the sample application to an Azure App Service web app by using Azure Pipelines in the subsequent steps. For now, replace the placeholder text {your-app-name} in the previous XML snippet with a unique name that you want to provide to the App Service web app. You'll then use this same name to create the web app.

   Important

   Don't include https or http in the sample application's URL.

4. Commit your changes to the main branch.

Set up Azure Pipelines access permissions for Azure

In this section, you'll configure your Azure DevOps project to have permissions to access the Azure Load Testing resource.

To access Azure resources, create a service connection in Azure DevOps and use role-based access control to assign the necessary permissions:

1. Sign in to your Azure DevOps organization (https://dev.azure.com/<yourorganization>).

2. Select Project settings > Service connections.

3. Select + New service connection, select the Azure Resource Manager service connection, and then select Next.

4. Select the Service Principal (automatic) authentication method, and then select Next.

5. Select the Subscription scope level, and then select the Azure subscription that contains your Azure Load Testing resource.

   

   You'll use the name of the service connection in a later step to configure the pipeline.

6. Select Save to create the connection.

7. Select the service connection from the list, and then select Manage Service Principal.

   

   You'll see the details of the service principal in the Azure portal. Note the service principal's Application (Client) ID value.

   

8. Assign the Load Test Contributor role to the service principal to allow access to the Azure Load Testing service.

   First, retrieve the ID of the service principal object. Select the objectId result from the following Azure CLI command:

   az ad sp show --id "<application-client-id>"
   

   Next, assign the Load Test Contributor role to the service principal. Replace the placeholder text <sp-object-id> with the ID of the service principal object. Also, replace <subscription-name-or-id> with your Azure subscription ID.

   az role assignment create --assignee "<sp-object-id>" \
       --role "Load Test Contributor" \
       --subscription "<subscription-name-or-id>"
   

Configure the Azure Pipelines workflow to run a load test

In this section, you'll set up an Azure Pipelines workflow that triggers the load test.

First, you'll install the Azure Load Testing extension from the Azure DevOps Marketplace, create a new pipeline, and then connect it to the sample application's forked repository.

1. Install the Azure Load Testing task extension from the Azure DevOps Marketplace.

   

   

2. In your Azure DevOps project, select Pipelines, and then select Create pipeline.

   

3. On the Connect tab, select GitHub.

4. Select Authorize Azure Pipelines to allow Azure Pipelines to access your GitHub account for triggering workflows.

5. On the Select tab, select the sample application's forked repository.

   

   The repository contains an azure-pipeline.yml pipeline definition file. You'll now modify this definition to connect to your Azure Load Testing service.

6. On the Review tab, replace the following placeholder text in the YAML code:

   Placeholder	Value
   <Name of your webapp>	The name of the Azure App Service web app.
   <Name of your webARM Service connection>	The name of the service connection that you created in the previous section.
   <Azure subscriptionId>	Your Azure subscription ID.
   <Name of your load test resource>	The name of your Azure Load Testing resource.
   <Name of your load test resource group>	The name of the resource group that contains the Azure Load Testing resource.

   Important

   The name of Azure web app should match the name that you used for the endpoint URL in the SampleApp.jmx test script.

   

7. Select Save and run, enter text for Commit message, and then select Save and run.

   

   Azure Pipelines now runs the CI/CD workflow. You can monitor the status and logs by selecting the pipeline job.

   

View results of a load test

For every update to the main branch, the Azure pipeline executes the following steps:

• Deploy the sample Node.js application to an Azure App Service web app. The name of the web app is configured in the pipeline definition.
• Trigger Azure Load Testing to create and run the load test, based on the Apache JMeter script and the test configuration YAML file in the repository.

To view the results of the load test in the pipeline log:

1. In your Azure DevOps project, select Pipelines, and then select your pipeline definition from the list.

2. Select the pipeline run to view the run summary.

   

3. Select Load Test in the Jobs section to view the pipeline log.

   

   After the load test finishes, you can view the test summary information and the client-side metrics in the pipeline log. The log also shows the URL to go to the Azure Load Testing dashboard for this load test.

4. In the pipeline log view, select Load Test, and then select 1 artifact produced to download the result files for the load test.

   

Define test pass/fail criteria

In this section, you'll add criteria to determine whether your load test passes or fails. If at least one of the pass/fail criteria evaluates to true, the load test is unsuccessful.

You can specify these criteria in the test configuration YAML file:

1. Edit the SampleApp.yml file in your GitHub repository.

2. Add the following snippet at the end of the file:

   failureCriteria: 
       - avg(response_time_ms) > 100
       - percentage(error) > 20
   

   You've now specified pass/fail criteria for your load test. The test will fail if at least one of these conditions is met:

   • The aggregate average response time is greater than 100 ms.
   • The aggregate percentage of errors is greater than 20%.

3. Commit and push the changes to the main branch of the repository.

   The changes will trigger the Azure Pipelines CI/CD workflow.

4. On the page for pipeline runs, select the most recent entry from the list.

   After the load test finishes, you'll notice that the pipeline failed because the average response time was higher than the number that you specified in the pass/fail criteria.

   

   The Azure Load Testing service evaluates the criteria during the test run. If any of these conditions fails, Azure Load Testing service returns a nonzero exit code. This code informs the CI/CD workflow that the test has failed.

5. Edit the SampleApp.yml file and change the test's pass/fail criteria:

   failureCriteria: 
       - avg(response_time_ms) > 5000
       - percentage(error) > 20
   

6. Commit the changes to trigger the Azure Pipelines CI/CD workflow.

   

   The load test now succeeds and the pipeline finishes successfully.

Pass parameters to your load tests from the pipeline

Next, you'll parameterize your load test by using pipeline variables. These variables can be secrets, such as passwords, or non-secrets.

In this tutorial, you'll reconfigure the sample application to accept only secure requests. To send a secure request, you need to pass a secret value in the HTTP request:

1. Edit the SampleApp.yaml file in your GitHub repository.

   Update the testPlan configuration setting to use the SampleApp_Secrets.jmx file:

   version: v0.1
   testName: SampleApp
   testPlan: SampleApp_Secrets.jmx
   description: 'SampleApp Test with secrets'
   engineInstances: 1
   

   The SampleApp_Secrets.jmx Apache JMeter script uses a user-defined variable that retrieves the secret value with the custom function ${__GetSecret(secretName)}. Apache JMeter then passes this secret value to the sample application endpoint.

2. Commit the changes to the YAML file.

3. Edit the config.json file in your GitHub repository.

   Update the enableSecretsFeature setting to true to reconfigure the sample application to accept only secure requests:

   {
       "enableSecretsFeature": true
   }
   

4. Commit the changes to the config.json file.

5. Edit the SampleApp_Secrets.jmx file.

6. Search for <stringProp name="HTTPSampler.domain">.

   You'll see three instances of <stringProp name="HTTPSampler.domain"> in the file.

7. Replace all three instances of the value with the URL of your sample web app:

   <stringProp name="HTTPSampler.domain">{your-app-name}.azurewebsites.net</stringProp>
   

   You'll deploy the sample application to an Azure App Service web app by using the GitHub Actions workflow. In the previous XML snippet, replace the placeholder text {your-app-name} with the unique name of the App Service web app.

   Important

   Don't include https or http in the sample application's URL.

8. Save and commit the Apache JMeter script.

9. Go to the Pipelines page, select your pipeline definition, and then select Edit.

   

10. Select Variables, and then select New variable.

11. Enter the Name (mySecret) and Value (1797669089) information. Then select the Keep this value secret checkbox to store the variable securely.

    

12. Select OK, and then select Save to save the new variable.

13. Edit the azure-pipeline.yml file to pass the secret to the load test.

    Edit the Azure Load Testing task by adding the following YAML snippet:

    secrets: |
      [
          {
          "name": "appToken",
          "value": "$(mySecret)"
          }
      ]
    

14. Save and run the pipeline.

    The Azure Load Testing task securely passes the secret from the pipeline to the test engine. The secret parameter is used only while you're running the load test, and then the value is discarded from memory.

Configure and use the Azure Load Testing task

This section describes the Azure Load Testing task for Azure Pipelines. The task is cross-platform and runs on Windows, Linux, or Mac agents.

You can use the following parameters to configure the Azure Load Testing task:

The following YAML code snippet describes how to use the task in an Azure Pipelines CI/CD workflow:

- task: AzureLoadTest@1
  inputs:
    azureSubscription: '<Azure service connection>'
    loadTestConfigFile: '< YAML File path>'
    loadTestResource: '<name of the load test resource>'
    resourceGroup: '<name of the resource group of your load test resource>' 
    secrets: |
      [
          {
          "name": "<Name of the secret>",
          "value": "$(mySecret1)",
          },
          {
          "name": "<Name of the secret>",
          "value": "$(mySecret1)",
          }
      ]
    env: |
      [
          {
          "name": "<Name of the variable>",
          "value": "<Value of the variable>",
          },
          {
          "name": "<Name of the variable>",
          "value": "<Value of the variable>",
          }
      ]

Clean up resources

If you don't plan to use any of the resources that you created, delete them so you don't incur any further charges. If you've deployed the sample application in a different resource group, you might want to repeat the following steps.

To delete resources by using the Azure portal:

1. Select the menu button in the upper-left corner, and then select Resource groups.

2. From the list, select the resource group that you created.

3. Select Delete resource group.

4. Enter the resource group name. Then select Delete.

To delete resources by using the Azure CLI, enter the following command:

az group delete --name <yourresourcegroup>

Remember, deleting the resource group deletes all of the resources within it.

Next steps

You've now created an Azure Pipelines CI/CD workflow that uses Azure Load Testing for automatically running load tests. By using pass/fail criteria, you can set the status of the CI/CD workflow. With parameters, you can make the running of load tests configurable.

• For more information about parameterizing load tests, see Parameterize a load test.
• For more information about defining test pass/fail criteria, see Define test criteria.