In my TechRepublic column, “Estimating the Unknown,” I noted that when organizations migrate from traditional to agile project management, one of the most challenging elements of transition is often the change in estimation techniques.

Comparison of estimation techniques

Estimating in traditional project management is usually task-based. The project manager, with the help of the team, develops a work breakdown structure (essentially a list of tasks), and then the subject matter experts take a stab at estimating the number of hours each task will take. Estimating in agile project management is typically feature-based, by which I mean that entire features, such as “find a flight-by source and destination city,” are estimated in their entirety.

In addition, while traditional, task-based estimating attempts to forecast an absolute number of hours for a task (or, at best, a range of hours), feature-based estimating is more focused on relative size of features, a sort of “small, medium, large” scale that refrains from attaching an absolute time prediction to each task.

Why bother switching estimation techniques?

Often, when I coach teams in their transition to agile, the question they ask me about switching from task-based to feature-based estimating is “Why bother?”

Why bother, they ask, when we’re all so familiar with task-based estimating, and our managers are accustomed to it? Why bother, when, after all this time, my team has finally gotten pretty good at it, and our estimates are usually fairly accurate? What exactly is the benefit of making the painful migration from the known to the unknown and from the predictive to the speculative?

Traditional project management

Let’s go back to our traditional planning scenario. Once the team has developed its task-based estimate, that estimate typically doesn’t change throughout the project. The actual time will change, as reality intrudes on our idealized estimates; this creates the familiar delta between planned and actual hours that can cause project managers to spend inordinate amounts of time accounting for the difference, explaining it to the client, and adjusting project plans, as tasks slip and delays cascade through the plan.

In traditional project management, plans and estimates are typically made once, in a “big bang,” front-loaded approach, and the rest of the project is spent adjusting to reality. Project managers, rather than working with their team to facilitate their creativity, to navigate obstacles, and to provide leadership and mentoring, are often found tucked away in an office somewhere, adjusting Gantt charts and task lists to conform to the reality of development.

Agile project management

In agile environments, every iteration is an opportunity to reconsider the plan and adjust to reality for the next iteration. It’s the realization of the famous aphorism that “a good plan now is better than a perfect plan tomorrow.” By creating a plan and estimate that’s good enough to get started with now, and with the realization that an accurate and predictive plan is an impossibility, we begin the march toward presenting a working product or prototype to the client immediately and enabling the project manager to play her proper role of leadership rather than that of project bureaucrat.

The central reason why it is important that agile planning be done at the feature level rather than the task level is that stakeholders and sponsors don’t care about tasks and often don’t understand what they mean. They are often seen as an impediment to transparency and communication rather than as an aid.

Stakeholders and sponsors pose a great danger of focusing the client on the granular details of the work rather than the business functionality required and can lead to unproductive, circular arguments that distract from the project and the atmosphere. I’ve gotten embroiled in numerous useless arguments with clients regarding task-level estimates:”Why is it going to take 6.5 hours to build the racks…last time it took 5.5 hours?”

Task-level decomposition also drives the project manager to manage at the task level rather than at the business-feature level, further diluting her role as a leader and facilitator and turning her into a foreman, counting “units of production” rather than results.

Not only do we estimate at different levels of granularity regarding the features we deliver, we also plan at different time levels as well. Many agile practitioners talk of the “planning onion,” with planning by day at the center, expanding out to planning at the iteration, release, and project level. The daily meeting, an integral element of agile practice, allows teams to plan in a very granular manner for the results that they plan to deliver that day, and, due to its immediacy, estimates for this work are much more likely to be accurate.

As we move further and further toward the horizon and as the work becomes more ambiguous, unknown, and prone to change, our planning is necessarily less precise. We don’t waste a lot of time planning, in artificial precision, for work that may never get done and that we really can’t estimate well anyway, since it holds so much uncertainty and mutability. “Estimating only what we can see” is realized in this multiple-time-horizon process and can save us, and our clients, from many hours of unrealistic predictions regarding project elements that can’t be known.

Obstacles to transitioning to new estimation techniques

The biggest obstacle to transitioning to new estimation techniques, in my experience, is the fact that clients, sponsors, stakeholders, and PMO officers have a high degree of comfort with traditional task-level, absolute-hour estimations. They usually recognize that they’re not really accurate and that the project plan will need to be adjusted continuously; they also acknowledge that many projects managed traditionally still come in over budget and past due. Still, change is always difficult, and familiarity can breed comfort rather than contempt. For these project sponsors, I often walk through the following scenario.

Let’s suppose that I’m building a data center with three experts: a facilities expert, a server expert, and a virtualization expert. I write a task plan that includes tasks for building and installing the racks, for building and installing the servers, and for virtualizing those servers.

To build my estimate, I’ll ask each subject matter expert for their opinion on the time required. The facilities expert may respond by saying, “It’ll take 20 hours,” giving her best guess and not building in any risk or contingency hours. The server expert, drawing on previous experience, may think, “It’ll take me 20 hours, but I’ll double that and say 40, since I don’t want to be embarrassed like the last project, when we got stuck with a DOA server that sucked up lots of extra time.” The virtualization expert may think, “It might take me 60 hours, but I’ll say 40 because I want to impress on the team how smart and competent I am.”

Now multiply this process by the 10 or 20 subject matter experts that a complex project can require, with each applying, not a dry technical calculation, but an emotional response that incorporates their desire to appear competent, to avoid embarrassment, or to hit a preconceived date imposed by the project manager or client. It’s pretty easy to see that while traditional estimating may be perceived as familiar and safe by many clients and PMOs this simple exercise is actually fraught with emotion, ego, and many other nontechnical elements that can render the result meaningless.

Does agile solve this problem?

Agile project managers and practitioners recognize human elements (emotion, ego, and desire) exist and by iteratively, incrementally planning and building the product that the client needs, we incorporate what we learn from every iteration in the plan for the next one, thus creating self-correcting, self-regulating plans and estimates that are based in reality. This concept of reality-based estimation and development, which enables the client to continuously correct our course and the team to constantly correct our plans and estimates, is at the heart of the agile philosophy.