Collecting metrics is the best way to know whether a process is under control and within overall tolerances. The testing process is no different. If you have a large or complex project, you should collect testing metrics to ensure that the process is going well and accomplishing its objectives.
As an example, one of the objectives of testing is to catch as many errors as possible. However, you should expect that the more testing you perform, the fewer errors you will catch over time. If your metrics show that you continue to uncover more and more errors over time, it probably means you have some fundamental development processes that are out of control.
The place to start
The collection of testing metrics starts after the unit testing. Unit testing is too unpredictable and too unstructured to make much sense from a metrics standpoint. However, the following metrics can be captured from integration testing through the end of the testing process.
- Total errors uncovered: This is strictly a count of each error uncovered. By itself, it does not have as much meaning as when it is combined with other information for analysis.
- Total errors by week: This measure gives an overall sense for how the testing process is going. You would expect that a vast number of errors would get caught in the early stages of testing. As time goes by, the number of errors uncovered per week should decrease. You may never reach a point were you find no more errors, but the team cannot complete testing if the number of errors is not tapering down close to zero.
- Number of hours spent correcting errors per week: This metric provides a sense for the relative complexity of errors caught in testing. Early in the testing process, you may encounter errors that are time consuming to correct. However, as the testing cycle proceeds, the time that you spend correcting errors should decrease.
- Error correction hours per error uncovered: This metric is similar to the prior metric. As you get further and further into the testing process, you should find that the time required to fix the typical error is getting shorter and shorter. If you uncover late errors that are time consuming to correct, it is usually a sign that the errors trace back to problems in the design or business requirements phases.
- Number of completed test cases/total number of test cases: This metric gives the team a sense for how much work has been completed and how much is remaining.
- Budget vs. actual effort, cost, and duration: These are standard metrics that point out how the project team is executing the testing process vs. the original estimates.
- Total testing cost/effort vs. total development cost/effort: This metric provides some perspective on how one project spends its development time vs. other projects. If the average project spends 25 percent of its time in testing and your project spent 35 percent in testing, it might point out certain inefficiencies.
Slicing and dicing the errors
Error tracking can be tracked in a number of ways. They are all ways to provide further information to help in making decisions on how the testing process is going and where to focus the team’s testing time.
- Errors by cause: The project team can track and categorize errors that are similar in nature. For instance, program errors will probably account for most errors in the beginning, but you may find later that errors can be traced to poor documentation or operator errors. Errors that are coming from a common cause can be attacked more vigorously.
- Errors by subsystem: If a small number of subsystems have an unusually high number of errors, it can be a sign that more focus needs to be placed in those particular areas.
Production defect metrics
Problems uncovered in the testing process are referred to as errors. Once the application goes into production status, these same problems are referred to as defects.
- Number of defects in production: This is an important metric to determine the overall effectiveness of the testing process. In a perfect world, an application would have no errors when it is implemented in production. However, there are always some. Counting these errors gives a sense of the stability of the system. You can capture similar metrics after implementation as you did during the testing process. For instance, you can capture the number of errors per week, which should decline over time, and the number of hours to fix production errors, which should also decline over time.
- Production downtime: This metric measures the amount of time an application is down because of errors. This can show the business pain caused by production defects.
- Cost of defects: This metric measures the cost of defects uncovered in production. This can include the cost of fixing the defects and the cost of people and resources that are idled or underutilized because of the downtime.
Measurement is the key
The only way to really know how your testing is going is to measure it. Here is what you should keep in mind as you keep track of your testing.
- Testing metrics should be captured to provide a sense for how the entire testing process is progressing.
- The most important component of testing metrics is the capturing of the number of errors. In addition, further information can be gathered as to the cost and effort associated with correcting the errors. With this information, valuable insight can be gathered into whether the testing process is under control or not.
- Much of the same information that is captured during the testing process can also be gathered after the application goes into production. At that point, problems should continue to be tracked as defects.