Quality Dashboard (Agile)

Article
02/04/2013

You can use the Quality dashboard to obtain an overview of progress occurring in the test, development, and build areas as they relate to the quality of the software under development. The team can use the Quality dashboard to learn and make decisions that support team goals around product quality.

By using this dashboard, you can review test progress, build states, progress in resolving and closing bugs, rate of bug reactivations, the percentage of code that has been tested, and the trends in code changes. Each of these metrics is plotted for the most recent four weeks.

Note

You access dashboards through your team project portal. You can access the Quality dashboard only if your that portal has been enabled and is provisioned to use Microsoft Office SharePoint Server 2007. For more information, see Dashboards (Agile) or Access a Team Project Portal and Process Guidance.

In this topic

Data That Appears in the Dashboard
Required Activities for Tracking Quality
Troubleshooting Quality Issues
Customizing the Quality Dashboard

You can use this dashboard to answer the following questions:

Is the test effort on track?
Is the team testing the appropriate functionality?
Are the team's bug fixes of high quality?
Are tests stale?
Does the team have sufficient tests?
Are any bottlenecks occurring?

Required Permissions

To view the dashboard, you must be assigned or belong to a group that has been assigned the Read permission in SharePoint Products for the team project. To modify, copy, or customize a dashboard, you must be assigned or belong to a group that has been assigned the Members permission in SharePoint Products for the team project. For more information, see Add Users to Team Projects.

To modify a report in Office Excel, you must be a member of the TfsWarehouseDataReaders security role in SQL Server Analysis Services and you must be assigned or belong to a group that has been assigned the Members permission in SharePoint Products for the team project. For more information, see Grant Access to the Databases of the Data Warehouse for Visual Studio ALM.

To view a work item, you must be a member of the Readers group or your View work items in this node permission must be set to Allow. To create or modify a work item, you must be a member of the Contributors group or your Edit work items in this node permission must be set to Allow. For more information, see Managing Permissions.

Data That Appears in the Dashboard

Team members can use the Quality dashboard to determine the overall quality of the product that they are developing. In an ideal case, test pass rates, bugs, and code churn all show the same picture, but often they do not. When you find a discrepancy, you must examine more closely the appropriate build and data series. The Quality dashboard combines the test results, code coverage from testing, code churn, and bugs, to help you understand many perspectives at the same time.

Specifically, this dashboard displays the Web parts that the following illustration shows and the following table describes.

Product Quality Dashboard

Note

The Test Plan Progress report is available only when the team creates test plans and runs tests by using Test Runner and Microsoft Test Manager. For information about how to define test suites and test plans, see Organizing Test Cases Using Test Suites.

Progress, build, and code charts, reports Step 1 through Step 6 , do not appear when the data warehouse for the team project is not available.

To learn more about how to interpret, update, or customize the charts that appear in the Quality dashboard, see the topics in the following table.

Web part	Data displayed	Related topic
	Stacked area graph of the test results of all Test Cases grouped by their most recent recorded outcome - Never Run, Blocked, Failed, or Passed - during the most recent four weeks.	Test Plan Progress Report
	Stacked columns that show how many builds Failed or Succeeded during the most recent four weeks.	Build Status Excel Report
	A stacked area graph of the cumulative count of all Bugs, grouped by state, during the most recent four weeks.	Bug Progress Excel Report
	A stacked area graph of how many bugs the team has reactivated from the resolved or closed state during the most recent four weeks.	Bug Reactivations Excel Report
	Line chart that shows the percentage of code tested by build verification tests (BVT) and other tests during the most recent four weeks.	Code Coverage Excel Report
	Stacked area graph that shows how many lines of code the team added, removed, and changed in the check-ins before the build during the most recent four weeks.	Code Churn Excel Report
	List of upcoming events. This list is derived from a SharePoint Web part.	Not applicable
	Count of active, resolved, and closed work items. You can open the list of work items by clicking each number. This list is derived from a Team Web Access Web part.	Work Items and Workflow (Agile)
	List of recent builds and their status. You can view more details by clicking a specific build. This list is derived from a Team Web Access Web part. Legend: : Build in Progress : Build Not Started : Build Succeeded : Build Failed : Build Stopped : Build Partially Succeeded	Manage and View Completed Builds
	List of the most recent check-ins. You can view more details by clicking a specific check-in. This list is derived from a Team Web Access Web part.	Using the Check In and Pending Changes Windows

Required Activities for Monitoring Quality

For the Quality Dashboard to be useful and accurate, the team must perform the activities that this section describes.

Required Activities for Tracking Test Plan Progress

For the Test Plan Progress report to be useful and accurate, the team must perform the following activities:

Define Test Cases and User Stories, and create Tested By links between Test Cases and User Stories.
Define Test Plans, and assign Test Cases to Test Plans. For more information, see Defining Your Testing Effort Using Test Plans.
For manual tests, mark the results of each validation step in the Test Case as passed or failed.

Important

Testers must mark a test step with a status if it is a validation test step. The overall result for a Test Case reflects the status of all the test steps that the tester marked. Therefore, the Test Case will have a status of failed if the tester marked any test step as failed or did not mark it.

For automated tests, each Test Case is automatically marked as passed or failed.
(Optional) To support filtering, assign Iteration and Area paths to each Test Case.

Note

For information about how to define area and iteration paths, see Create and Modify Areas and Iterations.

Required Activities for Tracking Bug Progress and Bug Reactivations

For the Bug Progress and Bug Reactivations reports to be useful and accurate, the team must perform the following activities:

Define Bugs.
Update the State of each Bug as the team fixes, verifies, closes, or reactivates it.
(Optional) Specify the Iteration and Area paths of each Bug if you want to filter by those fields.

Required Activities for Tracking Build Status, Code Coverage, and Code Churn

For the Build Status, Code Coverage, and Code Churn reports to be useful and accurate, team members must perform the following activities:

Configure a build system. To use Team Foundation Build, you must set up a build system.

For more information, see Configure Your Build System.
Create build definitions. You can create several build definitions and then run each of them to produce code for a different platform. Also, you can run each build for a different configuration.

For more information, see Define Your Build Process.
Define tests to run automatically as part of the build. As part of the build definition, you can define tests to run as part of the build or to fail if the tests fail.

For more information, see Define a Build Using the Default Template.
Configure tests to gather code coverage data. For code coverage data to appear in the report, team members must instrument tests to gather that data.

For more information, see How to: Configure Code Coverage Using Test Settings for Automated Tests and How to: Gather Code-Coverage Data with Generic Tests.
Run builds regularly. You can run builds at regular intervals or after every check-in. You can create regular builds when you use the schedule trigger.

For more information, see Create a Basic Build Definition and Run and Monitor Builds.

Note

Although a team member can manually rate a build by using Build Explorer, this rating is not reflected in the Build Quality Indicators report. The build rating appears in the Build Summary report. For more information, see Rate the Quality of a Completed Build and Build Summary Report.

Troubleshooting Quality Issues

The following table describes specific quality issues that the Quality dashboard can help you monitor and identify actions the team can take.

Issue	Reports to review	Troubleshooting notes
Build failures	Builds Status	A nightly build is the heartbeat of software development projects. When builds are not completing successfully or are not passing build verification tests (BVT), the team must fix the problem immediately.
Tests failing	Test Plan Progress Code Churn	When the rates of failed tests and code churn are high, the team may investigate why the software is failing so often. Causes may include loose development practices or tests that are too rigorous for an early iteration cycle.
Tests passing but with a high rate of finding Bug	Test Plan Progress Bug Progress	When , many tests pass in the same period as many Bugs are found, the team might investigate the following possibilities: Tests might not be sufficiently rigorous for the current product stage. In early iterations, simple tests are good. However, tests should exercise broader scenarios and integration as the product matures. Tests might be stale or testing the wrong functionality. Different test techniques might offer better results. Bugs are being reported but not subject to testing. When Bugs are reported and are not linked to a Test Case, they are not subject to regression testing.
Tests are stale	Test Plan Progress Code Coverage Code Churn	When many tests pass, a significant amount of code changes, and code coverage decreases, the team might not be running tests that exercise the new code. Because tests are not developed at the same rate as the code changes, test coverage might become less and less adequate.
Team is not testing, closing, or reactivating resolved Bugs	Bug Progress	When a bulge occurs in the Bug Progress report for resolved Bugs, idevelopers are resolving Bugs, but testers have not verified and closed them. The team should investigate why this pattern has developed.
Too little testing	Test Plan Progress Code Churn	When the team is running few tests, code churn is high, and code coverage is less than expected, the team might need to allocate more resources to testing. In addition, the team should ensure that testers are focusing on the same functions as the rest of the team.
Reactivations	Bug Reactivations	When the team reactivates Bugs at a high or increasing rate, testers are frequently rejecting developers' fixes. The team must address these problems to avoid allocating significant resources toward reworking the rejected fixes. Potential causes include poor bug reporting, poor test lab management, or overly aggressive triage.
Inadequate unit testing	Code Coverage Code Churn	When a decrease in code coverage coincides with an increase in code churn, developers might be checking in code without any corresponding unit tests to cover it. In most cases, the code coverage should approach 100% if the team practices test-driven development or similar techniques. If unit tests are reused as BVTs, the code coverage should appear in the corresponding reports.

Customizing the Quality Dashboard

You can customize the Quality dashboard in the following ways:

Change the filters of each Excel report to focus on specific product areas or iterations.
Add a custom query Web part that displays the list of work items that the query finds. For example, you can add a query that lists all active Bugs that are not linked to a Test Case. This query will show the volume of Bugs that are being reported but that are not found through testing and therefore not subject to regression testing.
Add existing Excel reports, such as Bug Trends and Failure Analysis, to the dashboard.

For more information about how to work with and customize reports in Office Excel, see the following pages on the Microsoft Web site: