Tutorial: Identify performance regressions with GitHub Actions and Azure Load Testing Preview

Article
12/01/2021
10 minutes to read

In this tutorial, you'll learn how to automate performance regression testing with Azure Load Testing Preview and GitHub Actions. You'll configure a GitHub Actions CI/CD workflow to run a load test for a sample web application, and then use the results to identify performance regressions.

If you're using Azure Pipelines for your CI/CD workflows, see the corresponding Azure Pipelines tutorial.

You'll learn how to:

Set up your repository with files required for load testing.
Set up a GitHub workflow to integrate with Azure Load testing.
Run the load test and view results in the workflow.
Define test criteria for the load test to pass or fail based on thresholds.
Parameterize a load test using GitHub secrets.

Important

Azure Load Testing is currently in PREVIEW. See the Supplemental Terms of Use for Microsoft Azure Previews for legal terms that apply to Azure features that are in beta, preview, or otherwise not yet released into general availability.

Prerequisites

An Azure account with an active subscription. If you don't have an Azure subscription, create a free account before you begin.
A GitHub account where you can create a repository. If you don't have one, you can create one for free.
An existing Azure Load Testing resource. To create a Load Testing resource, see Create and run a load test.

Set up your repository

To get started, you first need a GitHub repository with the sample web application. You'll then use this repository to configure a GitHub Action workflow to run the load test.

Open a browser and navigate to the sample application's source GitHub repository: https://github.com/Azure-Samples/nodejs-appsvc-cosmosdb-bottleneck.git

The sample application is a Node.js app consisting of an Azure App Service web component and a Cosmos DB database.
Select Fork to fork the sample application repository to your GitHub account.

Configure the Apache JMeter script

The sample application's source repo includes an Apache JMeter script named SampleApp.jmx. This script carries out three API calls on each test iteration:

add - Carries out a data insert operation on Cosmos DB for the number of visitors on the webapp.
get - Carries out a GET operation from Cosmos DB to retrieve the count.
lasttimestamp - Updates the timestamp since the last user went to the website.

In this section, you update the Apache JMeter script with the URL of your sample web app.

In your sample application repository, open SampleApp.jmx for editing.
Search for <stringProp name="HTTPSampler.domain">.

You'll see three instances of <stringProp name="HTTPSampler.domain"> in the file.
Replace the value with the URL of your sample application's App Service in all three instances.
```
<stringProp name="HTTPSampler.domain">{your-app-name}.azurewebsites.net</stringProp>
```
You'll deploy the sample application to an Azure App Service web app by using the GitHub Actions workflow in the subsequent steps. For now, replace the placeholder text {your-app-name} in the previous previous XML snippet with a unique name you wish to provide to the App Service web app. You'll then use this same name to create the web app.

Important

Don't include https or http in the sample application's URL.
Commit your changes to the main branch.

Set up GitHub access permissions for Azure

In this section, you'll configure your GitHub repository to have permissions for accessing the Azure Load Testing resource.

To access Azure resources, you'll create an Azure Active Directory service principal and use role-based access control to assign the necessary permissions.

Run the following Azure CLI command to create a service principal:

az ad sp create-for-rbac --name "my-load-test-cicd" --role contributor \
                         --scopes /subscriptions/<subscription-id> \
                         --sdk-auth

In the previous command, replace the placeholder text <subscription-id> with your Azure subscription ID of your Azure Load Testing resource.

The outcome of the Azure CLI command is a JSON string, which you'll add to your GitHub secrets in a later step.

{
  "clientId": "<my-client-id>",
  "clientSecret": "<my-client-secret>",
  "subscriptionId": "<my-subscription-id>",
  "tenantId": "<my-tenant-id>",
  (...)
}

Navigate to your forked sample application GitHub repository.
Add a new secret to your GitHub repository by selecting Settings > Secrets > New repository secret.
Enter AZURE_CREDENTIALS for the Name, paste the JSON response from the Azure CLI for the value, and then select Add secret.
To authorize the service principal to access the Azure Load Testing service, assign the Load Test Contributor role to the service principal.

First, retrieve the service principal object ID by running this Azure CLI command:
```
az ad sp list --filter "displayname eq 'my-load-test-cicd'" -o table
```
Next, assign the Load Test Contributor role to the service principal. Replace the placeholder text <sp-object-id> with the ObjectId value from the previous Azure CLI command. Also, replace the <subscription-name-or-id> with your Azure subscription ID.
```
az role assignment create --assignee "<sp-object-id>" \
    --role "Load Test Contributor" \
    --subscription "<subscription-name-or-id>"
```

Configure the GitHub Actions workflow to run a load test

In this section, you'll set up a GitHub Actions workflow that triggers the load test.

To run a test with Azure Load Testing from a CI/CD workflow, you need a load test YAML configuration file. The sample application repository contains the SampleApp.yaml file that contains the parameters for running the test.

Open the .github/workflows/workflow.yml GitHub Actions workflow file in your sample application repository.

Edit the file and replace the following placeholder texts.

Placeholder	Value
`<your-azure-web-app>`	The Azure App Service web app name. This name should match the name used for the endpoint URL in the SampleApp.jmx test script.
`<your-azure-load-testing-resource-name>`	The name of your Azure Load Testing resource.
`<your-azure-load-testing-resource-group-name>`	The resource group name that contains the Azure Load Testing resource.

Important

The name of Azure web app should match the name you used for the endpoint URL in the SampleApp.jmx test script.

env:
  AZURE_WEBAPP_NAME: "<your-azure-web-app>"
  LOAD_TEST_RESOURCE: "<your-azure-load-testing-resource-name>"
  LOAD_TEST_RESOURCE_GROUP: "<your-azure-load-testing-resource-group-name>"

Commit your changes directly to the main branch.

The commit will trigger the GitHub Actions workflow in your repository. You can verify that the workflow is running by navigating to the Actions tab.

View load test results

The GitHub Actions workflow executes the following steps for every update to the main branch:

Deploy the sample Node.js application to an Azure App Services web app. The name of the web app is configured in the workflow file.
Trigger Azure Load Testing to create and run the load test based on the Apache JMeter script and the test configuration YAML file in the repository.

In this section, you'll view the load test results in the GitHub Actions workflow log.

Select the Actions tab in your GitHub repository to view the list of workflows runs.
Select the workflow run from the list to navigate to the run details and logging information.

Once the load test finishes, you can view the test summary information and the client-side metrics in the workflow logs. The log also shows the steps to navigate to the Azure Load Testing dashboard for this load test.
In the workflow run details screen, select the loadTestResults artifact to download the load test result files.

Define test pass/fail criteria

In this section, you'll add failure criteria to determine the outcome of your load test. If at least one of the failure criteria evaluates to true, the load test is unsuccessful.

You can specify these criteria in the test configuration YAML file.

Edit the SampleApp.yml file in your GitHub repository.
Add the following snippet at the end of the file.
```
failureCriteria: 
- avg(response_time_ms) > 100
- percentage(error) > 20
```
You've now specified failure criteria for your load test. The test will fail if at least one of these conditions is met:
- The aggregate average response time is greater than 100 ms.
- The aggregate percentage of errors is greater than 20%.
Commit and push the changes to the main branch of the repository.

The changes will trigger the GitHub Actions CI/CD workflow.
Select the Actions tab, and then select the most recent workflow run to view the workflow logs.

After the load test finishes, you'll notice that the workflow failed because the average response time was higher than you specified in the failure criteria.

The Azure Load Testing service evaluates the criteria during the test execution. If any of these conditions fails, Azure Load Testing service returns a non-zero exit code. This code informs the CI/CD workflow that the test has failed.

Edit the SampleApp.yml file and change the test failure criteria.

failureCriteria: 
- avg(response_time_ms) > 5000
- percentage(error) > 20

Commit the changes to trigger the GitHub Actions workflow.

The load test now succeeds and the workflow finishes successfully.

Pass parameters to your load tests from the workflow

Next, you'll parameterize your load test using workflow variables. These parameters can be secrets such as passwords or non-secrets.

In this tutorial, you'll reconfigure the sample application to only accept secure requests. To send a secure request, you'll need to pass a secret value in the HTTP request.

Edit the SampleApp.yaml file in your GitHub repository.

Update the testPlan configuration setting to use the SampleApp_Secrets.jmx file.
```
version: v0.1
testName: SampleApp
testPlan: SampleApp_Secrets.jmx
description: 'SampleApp Test with secrets'
engineInstances: 1
```
The SampleApp_Secrets.jmx Apache JMeter script uses a user-defined variable that retrieves the secret value with the custom function ${__GetSecret(secretName)}. Apache JMeter then passes this secret value to the sample application endpoint.
Commit the changes to the YAML file.
Edit the config.json file in your GitHub repository.

Update the enableSecretsFeature setting to true to reconfigure the sample application to only accept secure requests.
```
{
    "enableSecretsFeature": true
}
```
Commit the changes to the config.json file.
Edit the SampleApp_Secrets.jmx file.
Search for <stringProp name="HTTPSampler.domain">.

You'll see three instances of <stringProp name="HTTPSampler.domain"> in the file.
Replace the value with the URL of your sample application's App Service in all three instances.
```
<stringProp name="HTTPSampler.domain">{your-app-name}.azurewebsites.net</stringProp>
```
You'll deploy the secure sample application to an Azure App Service web app by using the GitHub Actions workflow in subsequent steps. In the previous XML snippet, replace the placeholder text {your-app-name} with the unique name of the App Service web app. You'll then use this same name to create the web app.

Important

Don't include https or http in the sample application's URL.
Save and commit the Apache JMeter script.
Add a new secret to your GitHub repository by selecting Settings > Secrets > New repository secret.
Enter MY_SECRET for the Name, and 1797669089 for the value, and then select Add secret.
Edit the .github/workflows/workflow.yml file to pass the secret to the load test.

Edit the Azure Load Testing action by adding the following YAML snippet.
```
secrets: |
  [
      {
      "name": "appToken",
      "value": "${{ secrets.MY_SECRET }}"
      }
  ]
```
Commit the changes, which trigger the GitHub Actions workflow.

The Azure Load Testing task securely passes the repository secret from the workflow to the test engine. The secret parameter is only used while running the load test, and then its value is discarded from memory.

Azure Load Testing Action

This section describes the Azure Load Testing GitHub action. You can use this action by referencing azure/load-testing@v1 action in your workflow. The action runs on Windows, Linux, and Mac runners.

You can use the following parameters to configure the GitHub action.

Parameter	Description
`loadTestConfigFile`	Required Path to the load test YAML configuration file. The path is fully qualified or relative to the default working directory.
`resourceGroup`	Required Name of the resource group that contains the Azure Load Testing resource.
`loadTestResource`	Required Name of an existing Azure Load Testing resource.
`secrets`	Array of JSON objects that consist of the name and value for each secret. The name should match the secret name used in the Apache JMeter test script.
`env`	Array of JSON objects that consist of the name and value for each environment variable. The name should match the variable name used in the Apache JMeter test script.

The following YAML code snippet describes how to use the action in a GitHub Actions workflow.

- name: 'Azure Load Testing'
  uses: azure/load-testing@v1
  with:
    loadTestConfigFile: '< YAML File path>'
    loadTestResource: '<name of the load test resource>'
    resourceGroup: '<name of the resource group of your load test resource>' 
    secrets: |
      [
          {
          "name": "<Name of the secret>",
          "value": "${{ secrets.MY_SECRET1 }}",
          },
          {
          "name": "<Name of the secret>",
          "value": "${{ secrets.MY_SECRET2 }}",
          }
      ]
    env: |
      [
          {
          "name": "<Name of the variable>",
          "value": "<Value of the variable>",
          },
          {
          "name": "<Name of the variable>",
          "value": "<Value of the variable>",
          }
      ]

Clean up resources

Important

You can reuse the Azure Load Testing resource that you created for other Azure Load Testing tutorials and how-to articles.

If you don't plan to use any of the resources that you created, delete them so you don't incur any further charges. If you've deployed the sample application in a different resource group, you might want to repeat the following steps.

In the Azure portal:
1. Select the menu button in the upper-left corner, then select Resource groups.
2. From the list, select the resource group that you created.
3. Select Delete resource group.
4. Enter the resource group name. Then select Delete.
Alternately, you can use the Azure CLI.
```
az group delete --name <yourresourcegroup>
```
Remember, deleting the resource group deletes all of the resources within it.

Next steps

You've now created a GitHub Actions workflow that uses Azure Load Testing for automatically running load tests. By using pass/fail criteria, you can set the status of the CI/CD workflow. With parameters, you can make the load test execution configurable.

For more information about parameterizing load tests, see Parameterize a load test.
For more information about defining test pass/fail criteria, see Define test criteria.

Tutorial: Identify performance regressions with GitHub Actions and Azure Load Testing Preview

Please rate your experience

Prerequisites

Set up your repository

Configure the Apache JMeter script

Set up GitHub access permissions for Azure

Configure the GitHub Actions workflow to run a load test

View load test results

Define test pass/fail criteria

Pass parameters to your load tests from the workflow

Azure Load Testing Action

Clean up resources

Next steps

Feedback