Feedback Edit

Define pass/fail criteria for load tests by using Azure Load Testing Preview

  • Article
  • 2 minutes to read

In this article, you'll learn how to define pass/fail criteria for your load tests with Azure Load Testing Preview.

By defining test criteria, you can specify the performance expectations of your application under test. By using the Azure Load Testing service, you can set failure criteria for various test metrics.

Important

Azure Load Testing is currently in preview. For legal terms that apply to Azure features that are in beta, in preview, or otherwise not yet released into general availability, see the Supplemental Terms of Use for Microsoft Azure Previews.

Prerequisites

  • An Azure account with an active subscription. If you don't have an Azure subscription, create a free account before you begin.
  • An Azure Load Testing resource. If you need to create an Azure Load Testing resource, see the quickstart Create and run a load test.

Load test pass/fail criteria

This section discusses the syntax you use to define Azure Load Testing pass/fail criteria.

You use Aggregate_function (client_metric) condition value syntax. When a criterion evaluates to true, the load test gets the failed status.

Parameter Description
Client metric Required. The client metric on which the criteria should be applied.
Aggregate function Required. The aggregate function to be applied on the client metric.
Condition Required. The comparison operator.
Threshold Required. The numeric value to compare with the client metric.
The threshold evaluates against the aggregated value.

Load Testing supports the following combination of parameters:

Metric Aggregate function Threshold Condition
response_time_ms avg (average) Integer value, representing number of milliseconds (ms) > (greater than)
error percentage Numerical values in the range 0-100, representing a percentage > (greater than)

Define test pass/fail criteria in the Azure portal

In this section, you configure test criteria for a load test in the Azure portal.

  1. In the Azure portal, go to your Azure Load Testing resource.

  2. On the left pane, select Tests to view the list of load tests, and then select the test you're working with.

    Screenshot of the 'Configure' and 'Test' buttons and a list of load tests.

  3. Select the Test criteria tab.

    Screenshot that shows the 'Test criteria' tab and the pane for configuring the criteria.

  4. On the Test criteria pane, use the dropdown lists to select the Metric, Aggregate function, Condition, and Threshold values for your test.

    Screenshot of the 'Test criteria' pane and the dropdown controls for adding test criteria to a load test.

    You can define a maximum of 10 test criteria for a load test. If there are multiple criteria for the same client metric, the criterion with the lowest threshold value is used.

  5. Select Apply to save the changes.

When you run the load test, Azure Load Testing uses the updated test configuration. The test run dashboard shows the test criteria and indicates whether the test results pass or fail the criteria.

Screenshot that shows the test criteria on the load test dashboard.

Define test pass/fail criteria in CI/CD workflows

In this section, you learn how to define load test pass/fail criteria for continuous integration and continuous delivery (CI/CD) workflows. To run a load test in your CI/CD workflow, you use a YAML test configuration file.

  1. Open the YAML test configuration file.

  2. Add the test criteria to the configuration file. For more information about YAML syntax, see test configuration YAML reference.

    failureCriteria: 
    - avg(response_time_ms) > 300
    - percentage(error) > 20

  3. Save the YAML configuration file.

When the CI/CD workflow runs the load test, the workflow status reflects the status of the pass/fail criteria. The CI/CD logging information shows the status of each of the test criteria.

Screenshot that shows the test criteria in the CI/CD workflow log.

Next steps