Leadership

How to use project data to develop a better estimation matrix

Dr. Andrew Makar discusses effort estimation and shares his efforts to develop a better estimation matrix based on actual project data.

During lessons learned sessions, a common practice is to highlight what went well and identify areas for improvement on a project. Agile enthusiasts acknowledge project teams should conduct retrospectives (a.k.a. lessons learned sessions) after every release and iteration. In waterfall projects, this activity usually occurs at the end of the project. Regardless of your preferred methodology, I recommend comparing actual duration and actual effort to the baselined effort as part of a lessons learned session.

Effort estimation

Project teams often spend time at the end of a project documenting lessons learned, although few measure their actual performance and record it for future estimation.

During effort estimation, project teams conduct either a bottom-up or a top-down estimate. Bottom-up estimates require a significant investment in time to define scope and build an accurate estimate. Top-down approaches use analogous estimates and rely on expert opinion to estimate duration at a high level. If you start collecting actual performance data and compare actual results against baseline estimates, you can achieve an analogous estimation tool that is based on bottom-up data from past projects.

Estimation matrix

In my system implementations, a common practice involved developing an estimation matrix that categorized reports, interfaces, conversion, enhancements, and forms (or screens) and assigning a complexity level of low, medium, or high. Low estimates were assigned 8 hours of effort; medium estimates were assigned 16 to 24 hours of effort; and high estimates were assigned 32 to 80 hours of effort. The range was tweaked based on the information available. The key benefit was it provided a starting point for estimation based on basic information.

Over the past few years, I started collecting metrics against the Microsoft Project schedule so I could develop a better estimate matrix based on actual project data.

Figure A depicts a custom table that I used to track and categorize the actual project data for future comparison. In this table, I added custom fields to categorize the deliverables, assign a complexity rating, and include a flag to include in my estimation analysis. Figure A

MyEstimation table. (Click the image to enlarge.)
Figure B includes a close-up view of the interface analysis. Figure B

Interface actual and estimate comparison. (Click the image to enlarge.)

In this section, I can quickly see where I underestimated the code phases of specific interfaces. Interface 1 had a baseline duration of 2 days, and it took 6 days to complete the work. Since the data is recorded in days or hours, I can refine my estimates using both units. By examining the actual task data, I can start building better estimation metrics. It helps to understand the root cause of why a medium interface moved from 2 days to 6 days. Based on the root cause, I may adjust the matrix to accommodate better estimation.

After I export the data to Excel, I can update my matrix in Figure C. Figure C

Estimation matrix

Summary

For a lessons learned session, I am not suggesting you track every task, although I do recommend tracking against the major activities required to produce specific IT deliverables. For your next project, you'll need to conduct similar effort estimation activities. By comparing estimates against actuals for specific deliverables, you continue to build a better estimation matrix.

In my next column, I'll show how to track and export this data using a variety of views and maps in Microsoft Project.

How do you track your estimates and build more objective estimation tools? Let me know in the forums.

Get weekly PM tips in your inbox TechRepublic's IT Project Management newsletter, delivered on Wednesday, offers tips to help keep project managers and their teams on track. Automatically sign up today!

About

Dr. Andrew Makar is an IT program manager and is the author of How To Use Microsoft Project and Project Management Interview Questions Made Easy. For more project management advice visit http://www.tacticalprojectmanagement.com.

13 comments
maryanne.moore
maryanne.moore

I go a little bit deeper. I have the team estimate all the major tasks and set up my baseline. We track time by tasks throughout the project, which I keep in a spreadsheet and in MS Project. At the end, or any time during the project I know all the metrics about my projects. These are usually very large software development projects using a modified waterfall methodology. MA Moore

Tony Hopkinson
Tony Hopkinson

20 - 40 of the changes we make are kludge and bodge. You're last estimate, totally usless when it catches up with you. And it always does.... Without fail. Ever.....

raj_family
raj_family

Excellent Article. Thank you for the insight.

jfbauer
jfbauer

Good post and definitely looking forward to the next. I recently joined a smaller organization in legal services with a heavy custom software development focus. Their was no real strong methodology in place let alone project management discipline. It has taken some time, but I now have the developers completing a simple XLS for providing estimates that I small team created (more buy in since they created it) and MS Project for forecasting, we are finally reaching a point where estimates can be compared to actuals and time allocations can be measured against each line of business to help them better prioritize their IT efforts. Related blog: http://bit.ly/Ya8TQ

peter
peter

Nice article, it is a method I have developed since my first job too. In that first job I use to adopt a 0.6, 6 or 60 days method to any ideas that came up in discussion. It may have not been the best method but the colleagues I worked with at least knew if something was harder or easier to do by giving a definitive time scale to it. It was also easily understood by colleagues outside of the project team or working on applications when they wanted changes and managed their expectation of their change request actually happening.

Biggeorge0
Biggeorge0

Now then Tony, if you're using the kludge and bodge technique, then the least of the worries is estimation. Probably best to go back to the original design or re-check the staff / software capabilities.

PMPsicle
PMPsicle

I agree, a very good intro article on a very complex and confusing topic. The method used was simple but reasonably effective without becoming too precise to be useful (which itself isn't easy given that the individuals working on the project can cause the actual to vary by up to 500%). This last point is a key element in the real world when determining the number of segments estimates can be slotted into. One point that was made quietly was "It helps to understand the root cause of why a medium interface moved from 2 days to 6 days." I suggest that this point should not have been squeezed in quietly. It needs to be shouted. There are three key reasons for estimates and actuals to vary. The first is natural variance (i.e. I say it should take 1 day but by the time you finish getting your desk clear it's a 2 day job). The second is mis-understanding of complexity ... (I figured this is a moderate task that will take 2 days, but it had a calculation that took 2 days just to chase down.). Both of these are estimation variances - one manageable, one not. The third reason is external aka scope creep. (I know we said just give him a simple table entry but there has to be a listing to check the original against). Not all scope creep is handled at the project change level (nor should it be) however, it does affect the variance between actual and estimate. The variance attributed to this issue can point to a serious cultural or management issue that needs to be addressed. Finally, using baselined estimates is good. However, it should be stated (rather than presumed) that scope changes at the project level result in changes to the baseline and that these changes require re-estimation of the changed result. Unfortunately, all too often scope changes at the project level frequently are accompanied by informal (i.e. quick) estimates of the delay (rather than the result) or are used to adjust the entire project closer to actual. Neither is acceptable for good project practices but both are encouraged by some cultures. However, overall a good introductory article on the practical application of theory. Glen Ford, PMP http://www.trainingnow.ca http://www.learningcreators.com/blog

wmjas.shaw
wmjas.shaw

The outlined approach for estimating is excellent, and still my favourite when I can get (a) agreement to use it (b) the time to prepare the estimates properly. Unfortunately, the effort/ duration/ complexity ratios are very difficult to quantify from real-world data. If you rely only on your own projects, it would take many years to compile reliable data. Gathering data from other people's projects is very difficult because they will only generate the data if there is a stong methodology in place in the organization that mandates compliance with measuring actuals vs forecast. But much of the complexity comes from the ever-shifting technology. Numbers generated from a C or C++ project are not going to be accurate for a Java project. Even the tools available to the development teams, the amount of training provided to the developers and the skill levels of the team members can skew the results significantly. But, like democracy, despite its failings it still seems to be the best system available.

Tony Hopkinson
Tony Hopkinson

Kludge and bodge are business techniques not technical ones. What original design? If you can find it, it's so far out of date, it's neither use nor ornamament, and how in Cthulu's name are you meant to know if it was actually used anyway. Check staff capabilities... Management capabilities might have more effect, but then onky if quality stopped being an optional extra dropped at the first sign, of an 'Oh ***k', that bodge we did last year.

QAonCall
QAonCall

To sharpen your estimation efforts, then understanding that your estimate is clouded by facts on the ground is the driver. From there, you take that set of stats and move them to a new container (estimating bucket). In the future, when a project that represents similar risks, stakeholders etc, you evaluate that project for resemblance to this new statistical model, and use them (your baseline) and the newer model (upper boundary). (If you really want to mature this, there is a method to add influencers to your model, which is even more targeted, but obviously adds more cost to manage) The point is not to take your model based on a change in events (assumptions at the start of the project) but to understand their relationships. CMM/CMMI and other models apply statistics in order to predict future performance. When your model introduces newer influences, as an estimator/modeler you need to decide how you adapt your modeling, to allow you a better method to be more exact, not necessary fluxuate your methodology with variables that are not always easy to discern. The goal should be consistency, not always being exactly right. Most IT managers would be more willing to accept you never deviate by more than 5% than they would be willing to accept you are wrong 5% of the time (Again, consistent predictability is a great thing). If that 5% of the time represents, or potentially represents, the most important initiative in the company?s history, would you be less or more willing to be accepting of it. Context matters. If you present that manager with 2 sets of numbers and say the project will definitely come in between these 2 numbers, depending on how many of these risks are realized, or you say I will hit this number if nothing bad happens, as planners and managers we would and should rather plan for worst case, and work it to best case (mitigate risks, validate assumptions, report accurately and as information is available). IMHO?

l.solorzano
l.solorzano

thus, is there a way to estimate those who skew the results up front? To gather information on their levels of skill, experience based on their own past then bring into the picture. And Yes each project has dynamics (C , C++, Java, then Network, etc.) as such we should begin, not prevent, from establishing real workable systems, which keep up with 'ever-shifting' technology... for actual use of OUR skills ... as PMs...

PMPsicle
PMPsicle

So I treated it as such ... especially given that a fairly limited estimation technique was used. Advance apology for the quotations ... "To sharpen your estimation efforts, then understanding that your estimate is clouded by facts on the ground is the driver." Absolutely true ... in fact, please repeat. Loudly. Frequently. At the top of your voice. As often as you can. With technological assistance if possible! "The goal should be consistency, not always being exactly right. Most IT managers more willing to accept never deviate by more than 5% than you are wrong 5% of the time (Again, consistent predictability is a great thing)." Unfortunately, what a manager is willing to accept and the choices they have are not necessarily in agreement. (Nor for that matter the choices they believe are available). I have met many manager who have set performance requirements of 5 or 10 % when their process is only capable of producing results in the 50-100% range. The sad reality is that project time is at best an unbound bell curve. It is the stage of the process which determines the width of the curve. If I want a 10% variation in results, I need to estimate at a point in the DLC where I can get 10% variation. If I estimate at a point where I can only get 20% variation, then demanding a 10% variation is foolish. Why? Because I will ALWAYS have a percentage of projects which go beyond the boundaries. (Remember this is an unbounded bell curve ... high centre point, low edges but no end to how far out the edge goes). In short, the real statement of variance is 95% of the projects will come in within 20% of the estimate. That's the best an estimation can do. A double variance. (Not the figures -- the statement). It's at that point that artificial bounds come in to play. Effectively, the process used (and the go/no go points), place a boundary around the estimate -- 100% of our projects are within 20% of estimate. Why? Because the remaining 5% are cancelled. "If that 5% of the time represents, or potentially represents, the most important initiative in the company?s history, would you be less or more willing to be accepting of it." As a rational, knowledgeable manager -- it depends. If the success or failure of the initiative is budget bound (i.e. I need to be within x% of the budget), then I'm definitely going to say yes. And I'm going to use an Agile PM process (most likely ... there are other considerations). If, on the other hand, the result defines success or failure (i.e. I don't care what it costs we need this). Then the answer is going to be no. And I'm far more likely to use a Waterfall PM process (same commment as Agile).

Meadowsong
Meadowsong

Great discussion. The original article is a fine basic primer for estimating effort for a direct-report project team. However, it must e considered that the number of layers between the ?top? of the top-down and the ?bottom? of the bottom-up approaches could greatly skew the reliability of analogous estimating techniques. Further, if a matrix organization favors a functional (or more commonly, multiple concurrent projects for key resources) rather than a strict projectized alignment, then effort estimates will rarely equate to duration estimates. Resource leveling and negotiated availability of critical resources are musts to ensure that schedules and quality levels are maintained.

Editor's Picks