Stabilization, the period between “Code Complete” and “Release to Manufacturing,” is a hard phase to estimate for any software project. Many factors come into play during this phase, and the sheer uncertainty surrounding them is enough to give any project manager reason to question his or her profession. While no software is likely to estimate the stabilization phase of a software project perfectly, you can at least make sure your estimates are based on good information.
Download this spreadsheet to help you capture some of the big variables in a stabilization phase. Using these captured numbers, you can make a ballpark estimate of how long each release within the stabilization will take. The spreadsheet takes into account the following numbers:
- Test case executions: The total number of test cases that must be run to verify your software has met the requirements laid out for it
- Minutes required to run the average test
- Minutes to report a bug: The time it takes to verify steps and capture the bug in your tracking system
- Bugs per failed test case: The average number of bugs generated per failed test case
- Test FTEs: The number of Full Time Equivalents you have running tests
- Minutes to fix the average bug
- Development FTEs: The number of developers you have fixing the bugs generated by the tests
- Bug Defer Rate: On average, the percentage of bugs generated that will be deferred because they are deemed of low priority or low severity
- Test Failure Rate: On average, the percentage of test cases run that will fail
- Working hours per day: The number of hours of work you can expect from your testers and developers (This is NOT derived by adding up the hours between their start and finish times and subtracting an hour for lunch! This is the number of hours you can reasonably expect real work.)
In this sheet, the yellow shaded cells are not editable, in that they are calculated by a formula. The green cells are editable.
The first step in using the sheet is to enter your assumptions about the above variables. Figure A shows a sample sheet with these assumptions filled in.
Once you’ve provided these values, the rest of the sheet will populate and provide you with several data points for both the testing phase and the bug fix phase of each Release Candidate (RC). Figure B shows a sample RC.
Figure B shows us RC 1 for our sample from Figure A. We have 2,000 tests to run. At 10 minutes per test, this gives us a 333-hour test run. Based on our Test Failure Rate of 30 percent (this organization is not very confident about its code quality!) we expect 600 failed test cases. Based on our Bugs Per Failed Case rate of 2, we calculate we’ll have 1,200 bugs. It will take 200 hours to record the bugs. The good news is that we have 10 testers, so the whole thing will only take 7.6 days (given 7 hours per day of working time).
Moving to the Bug Fix section, we see we have 1,200 bugs presented to development. We will defer 50 percent of them, so we have 600 bugs to fix. Doing so will take 300 hours of developer time spread across 10 developers for a duration of 4.286 days. The entire RC, including testing and bug fixes, is estimated to take 12 days.
These values are for RC 1, where it is assumed that the values from Figure A will be in force. RCs 2 through 10 allow you to edit the Test Failure Rate, Bug Defer Rate, Test FTE, and Dev FTE values for each RC so you can refine your model as you move through the RCs. Figure C shows an example of these RC sections of the sheet.
Here we see a slightly different layout. You can see the addition of the four editable fields mentioned above. Also, the Tests To Run value for RCs 2 through 10 come from the Bug Fix section of the previous RC. So for RC 2 the Tests To Run value is the Bugs To Fix value from RC1. Likewise, the Tests To Run value for RC 3 will be derived from the Bugs To Fix value from RC2 (90 in this case).
Tips for use
- Get an accurate value for Working Hours Per Day. If your resources work an eight-hour day, this number should be something like 7 or even 6.5 to take into account time spent helping others, answering e-mail, etc.
- Remember that many of these figures are averages, not absolute values.
- If you have a few developers who are doing the bulk of the bug fixes, you can manipulate the Dev FTE value for each RC to take this fact into account. If you have 10 developers, but 90 percent of the bugs are falling on one developer, then you might set the Dev FTE value down to something like 2 or 3 so your durations will be more realistic.
This sheet will not provide 100-percent accuracy for estimating your stabilization, but it will give you a start in making estimates. Further, these sheets, along with the actual values for each phase for your projects, will give you statistics to use for the next project. By looking back over these sheets, you can make the process more accurate over time.
Download this spreadsheet and customize it with your own formulas and fields. Let it be the start of your own estimating tool.