Emerging Tech

Agile project management: Estimating the unknown

Every conversation about agile project management eventually turns to the question of estimating. Rick Freedman explains why estimation in an agile environment is not as mysterious as many PMs think.

Experienced project managers know the first question clients ask on any new project is "How much?" followed closely by "How soon?" Whenever the subject of agile project management is discussed, it's inevitable that one question will be asked: "How do you provide the client or sponsors with an estimate when you have no complete requirements document, no signed-off specification, and no ratified schedule?" Opponents of agile methods, deeply committed to the structured and predictive techniques championed by the Project Management Institute (PMI) and the Software Engineering Institute (SEI) at Carnegie Mellon University, point to this difficulty as the nail in the coffin of agile methodologies. Even organizations that are eager to try agile methods often raise this question with trepidation. Whenever I write a column about agile methods, this question of estimation pops up in the comments without fail.

Before we explore estimation techniques in an agile environment, let's remember a fundamental fact: Even in highly predictive, mature project environments like those recommended by PMI and SEI, estimating is a serious challenge, and estimates developed (even with thorough requirements and multiple user signoffs) are frequently wrong. I note this so we don't begin our discussion of agile estimating under a misapprehension.

The questions often asked about agile estimating are just as valid when asked about predictive methods. We may have gathered and documented binders full of user requirements and gotten sponsors and stakeholders to sign them in blood, but users will still reserve the right to say "No, that's not it" when we show them our first prototype. The fact that we've got a 300-step project plan with activities scheduled to the hour doesn't mean that the work will occur as predicted. In short, when we challenge agile proponents to prove that iterative, incremental development methods can be estimated, we shouldn't be surprised if they turn right around and demand to see the spotlessly accurate estimates we've created in our predictive projects.

This is not to deny that there are benefits to crafting a complete specification and project plan up front and using that as a basis for estimating. Especially in projects that are similar to ones we've delivered previously, history can be a meaningful guide and can enable us to derive relatively accurate estimates based on tasks we've done before. As noted in my previous column, agile methods are most appropriate in innovative projects, which lack this historical record. It's in these inventive projects (without analogies of previous efforts to lean on) that the real challenge of estimation in agile programs emerges. Estimation is a challenge in any project effort, but in these never-before-seen initiatives, it can seem impossible.

In fact, it's not. Some simple concepts can make agile estimating much less intimidating than it seems. The first general principle, which is valid in both agile and predictive project environments, is "don't estimate further than you can see." When we apply an incremental development approach, projects are divided into a series of iterations; each iteration results in the delivery of a working prototype that the client can evaluate and then be modified or expanded. Most agile techniques call for each iteration to be both time bound and feature bound. Ahead of the development of each iterative prototype, the team and sponsor agree on how much time they'll take to deliver that iteration and which features will be delivered. Under that method, it becomes pretty obvious how to estimate the time required to deliver that iteration; it's the "time-box" (to use the common agile language) that's been established by mutual consent. That time-box is usually established by listing the features expected to be delivered in the iteration at hand and estimating how long it will take to deliver those features.

As an agile project begins, the set of features to be delivered initially is often confined to the most fundamental features required to validate the project concept, with the "bells and whistles" and the more advanced capabilities deferred until later iterations. So, from the long list of features documented as elements of the ultimate product, we select those features for the initial iterations that will demonstrate that the underlying concept is feasible, and that will allow the client to begin the "more of this, less of that" conversation that inevitably occurs when we expose the prototype to the sponsor community. By limiting the early iterations to this reduced feature set, we make it easier to agree on a time-box for those features. We're only estimating the iteration at hand, with the estimation of future iterations deferred until we've got a clearer idea of how closely our developed prototype matches the customer's vision and expectations. Then we repeat this process, iteration by iteration.

This doesn't answer the dilemma of budgeting for the entire project, which is often what sponsors expect. After all, the purpose of an estimate in the corporate world is to enable the sponsor to set aside a budget for the entire initiative, not just the current iteration. This estimating task is also not that baffling, either. As iterations are time-bound, entire projects are both time-bound and cost-bound. When a homebuyer asks a construction company to build a house, the obvious first question is, "How much are you willing to spend?" The next question is usually, "When do you expect to move in?" after which the development of plans and features can begin. The client sets the budget that regulates the features that will be included in the new home, from the most humble pre-fabricated trailer to the most luxurious mansion.

This analogy also applies in the agile world. Rather than specifying every feature that the sponsor dreams of and then (often imperfectly) estimating the entire cost from ground-breaking to move-in, the agile developer, like the home developer, starts with the sponsor's available budget and then determines what can be delivered within that budget, and the time-box that the budget implies. In agile projects, as in projects delivered in a predictive methodology, the developers must set reasonable expectations for the sponsor so they're not expecting the Taj Mahal on a bungalow budget. As TechRepublic member PMP'sicle noted on my last piece, the question in agile development is not "how much will it cost for all these features?" but instead "how many of these features can you include within this budget and time-box?"

Estimating IT development projects will always be a fraught activity; since the technology is complex, users frequently change their minds, and things often don't work out as planned. This is true whether the approach is agile or predictive. In agile engagements, we estimate incrementally just as we develop incrementally, and we do all this development and estimation within the bounds of an agreed budget and time-box that's established in collaboration with the client at the beginning of the effort.

Note: For a thorough investigation of agile estimating, see Mike Cohn's classic book Agile Estimating and Planning. Get weekly PM tips in your inbox TechRepublic's IT Project Management newsletter, delivered on Wednesday, offers tips to help keep project managers and their teams on track. Automatically sign up today!

About

Rick Freedman is the author of three books on IT consulting, including "The IT Consultant." Rick is an independent consultant and trainer, working, through his company Consulting Strategies Inc., to help agile teams and organizations understand agile...

24 comments
Too_Old_2_Play
Too_Old_2_Play

In 1985, I worked for a large insurance company implementing a policy processing solution. After 18 months, the application went live "on-time" and "on-budget" (Hurrah). Then for the next 3 years, the project was re-budgeted under the guise of "upgrades" to address functionality that wasn't addressed during the original timeframe. While my lead-in seems off track, my point is that everything being touted by Agile project management is no different than the political gamesmanship that was played in the prehistoric times known as the 80's! Come on sports fans, project management is a dry, repetitive and thankless function that, for some reason, thinks it needs to rebrand itself every so often to stay in the headlines alongside the latest HW or SW initiative. It's good to know that all I need to do is add the latest set of buzzwords to my 20 years of project management experience and I, too, can be an Agile project manager if I was so inclined.

re0w
re0w

Hi Rick, Your analogy with the house missed the point. If you wanted to build a house, you would have a fixed budget (i.e. cash you have + bank loan). If your builder tried to convince you to go down the Agile path, would you follow? If so, you would be very unwise. What happens when your money runs out and you don't have the less important features (e.g. carpet, appliances, doors)? As a client, you need a guarantee that everything you want will be delivered for the budget you have; otherwise, you will go elsewhere. A client of a software company is no different. What Agile is all about is the shifting of risk. With a 'predictive' methodology, the burden of risk is with the builder (software company). They are contractually obliged to meet the customer's outlined desired, for the estimated price, irrespective of what it really costs them. On the other hand, with the Agile methodology, the burden of risk is with the client. If the money runs out and the product is not complete, the software company can simply walk away. Whilst I agree that the client bearing the burden of risk is fairer (they are esentially getting what they paid for, and disreputable builders will be driven out of the market), I do not believe that we (in the software industry) are yet to a point where we are in control of the market. We are still very much reliant on clients coming to us, and (with a constant supply of start-ups) supply definitely does outstrip demand. What we need is a manner in which we can deliver via an Agile approach (rapid response, more savings), but bear the burden of risk ourselves (as software companies). The Parabios Approach is one example of an agile methodology that does this. - Requirements are rapidly elicited up front in crude detail - not enough to design from, but enough to get an estimate from. - An estimate is made from these requirements and contingency is added to it (for missed requirements and requirements that appear smaller than they are). - Each timebox is then implemented the way it normally would (some of the requirements are selected; they are fleshed out; design, development and testing are undertaken). - When requirements change, a cost impact is calculated and the client is responsible for this. Using this approach, we have reaped the best of both worlds and had a lot of success. - The client gets an estimate up front. - The software company bears the risk of the estimate they give. - The client bears the risk of changes. - Issues/misunderstandings are identified quickly and resolved with low cost impact. What we need to bear in mind is why Agile came about in the first place. The issue was that a mistake that would cost $1 to fix in requirements, might cost $1000 by the time it got to development. The cost of developing (or redeveloping) requirements has never been an issue (it is cheap), it is the activities that take place after this that cost. That is the fundamental point that most agile methodologies miss. Regards, Mark

azvonko
azvonko

Instead of answering to Saurondor 06/22/09 (because most important thoughts and arguments for that answer I already expressed in my previous posts, and I simply have not so much time at the moment to comment again all I'd like to from that post), let me try here to point out yet some additional spots inspired by the original item. I find that we still started discussion under some misapprehensions, though the initiator wanted to avoid that. - - - [i]"... when we challenge agile proponents to prove that iterative, incremental development methods can be estimated, we shouldn?t be surprised if they turn right around and demand to see the spotlessly accurate estimates we?ve created in our predictive projects." [/i] Misapprehension! I hope nobody has ever stated that any predictive method produce a "spotlessly accurate" estimates! Unfortunately, no one method does. - - - [i]"The first general principle, which is valid in both agile and predictive project environments, is ?don?t estimate further than you can see.? " [/i] Misapprehension! This I find as fundamentally misleading, because the clients often ask just that. They have project ideas and plans for a longer period of time, so they ask us to estimate feasibility, costs, timeframe, etc for the whole project of their mind! Ok, they usually are ready to accept also some rough estimation for a long lasting project, but to prepare even such an estimation (assuming you take care of CONSISTENCY), that needs much more than agile puts in its scope. This leads me to an impression that agile is mostly preferred by the technically oriented IT people (what seems quite understandable), whilst the more predictive approach is better valued by the business oriented people, whose primary focus is on budgeting and contracting the project (also understandable). (Here I'd like to emphasize that above mentioned question of project consistency in my opinion is one of the major flaws in agile concept, because of high risks in consistency of requirements and consistency of solutions delivered during the project. Working for many years with one of my previous clients, in circumstances of some kind of "forced agile" I learned very well this lesson.) - - - [i]"When we apply an incremental development approach, projects are divided into a series of iterations; each iteration results in the delivery of a working prototype that the client can evaluate and then be modified or expanded. Most agile techniques call for each iteration to be both time bound and feature bound. Ahead of the development of each iterative prototype, the team and sponsor agree on how much time they?ll take to deliver that iteration and which features will be delivered. Under that method, it becomes pretty obvious how to estimate the time required to deliver that iteration; it?s the ?time-box? (to use the common agile language) that?s been established by mutual consent. That time-box is usually established by listing the features expected to be delivered in the iteration at hand and estimating how long it will take to deliver those features." [/i] Just to say to younger fellows who started with agile: prototyping method is not invention of agile; predictive methods has been used it in the same way for decades to demonstrate and get (internal or external) agreement on risky solutions and details in the project; so here the difference between these two methods remains again, generally, only in the size of "time-box". Besides, I remind that the same technique of partial deliveries has been used for the same long time under the name of "releases" or "versions", but what's important here is that these partial deliveries were developed in accordance with some general, long range plan ? roadmap ? for which was prepared all necessary estimations ahead the development phase. And ? guess what: while the project was going on, this roadmap has been constantly updated with new needs, ideas, corrections, changes, estimations, ...! Therefore, predictive methods do not deny partial delivery techniques, but insist on much more consistency of these deliveries! And that means more quality and lower total cost at the end. - - - [i]"As an agile project begins, the set of features to be delivered initially is often confined to the most fundamental features required to validate the project concept, with the ?bells and whistles? and the more advanced capabilities deferred until later iterations. So, from the long list of features documented as elements of the ultimate product, we select those features for the initial iterations that will demonstrate that the underlying concept is feasible, and that will allow the client to begin the ?more of this, less of that? conversation that inevitably occurs when we expose the prototype to the sponsor community. By limiting the early iterations to this reduced feature set, we make it easier to agree on a time-box for those features." [/i] That's right, absolutely agreed so far, that's the same technique as in predictive method. But the difference remains in what follows (as already said above). [i]"We?re only estimating the iteration at hand, with the estimation of future iterations deferred until we?ve got a clearer idea of how closely our developed prototype matches the customer?s vision and expectations. Then we repeat this process, iteration by iteration." [/i] - - - [i]"When a homebuyer asks a construction company to build a house, the obvious first question is, ?How much are you willing to spend?? The next question is usually, ?When do you expect to move in?? after which the development of plans and features can begin. The client sets the budget that regulates the features that will be included in the new home, from the most humble pre-fabricated trailer to the most luxurious mansion. "[/i] Misapprehension -- the fundamental one! In my experience the client usually would like to know first how much will it cost for all features he'd like to have (supposing that we are talking about a client who knows what he needs and wishes). And only later, when he realizes that he doesn't have money for all those needs and wishes, he comes to question on what he's able to get for his money. Therefore, it means that for the first answer to the client we'd have to prepare a reliable enough input specification as well as output calculation (for the entire project of his needs/wishes). I repeat: not necessarily specifying each single detail, but surely specifying all the details estimated to have a significant impact on the final delivery. (I can say that I am right now in such a situation with one of my clients. That means a lot of pre-contract work for both of us, since we both try not to overlook any bigger issue in estimation of a complex project which will take place for the next four years, and for which the budgeting is predefined. Ok, project will have some reserve for possible slight changes in the requirements during the development, but, of course, this amount is rather limited.) I agree, a different story might be if you had a client who does not know exactly what he wants, but he only knows he needs "a new house". In that scenario the first question to him really might be "How much are you willing to spend?", etc. But I think that's not a typical client in our business. Btw, it really makes me mad that sort of clients: their request sounds more like "Make me happy!", without any specification on what particularly might make them happy, so I feel like a court jester who has to make the king laughing. Needless to say that an honest approach with such a client usually takes of even more work to enlighten him about all possibilities, pros & cons, etc, so he becomes able to better articulate his wishes. - - - At the end, let's see an extreme example, just to emphasize the essence. Imagine you have a client who wants to build a settlement. That would be a multi-million dollars project, lasting for many years, and investor must approach very carefully to the project. So what would he need? I think, firstly the following: - To define the general plan of WHAT and HOW he would like to build, so the architects can choose the meaningful, consistent, optimal solutions on how and where to build the school, the church, the plant, apartments, roads, all the communication infrastructure, etc ? (remember: consistency!). - Then for sure he'd need the rough information on what the TOTAL COST would be for all that, and what would be some feasible END-TO-END TIME FRAME to build it, etc ? (remember: cost bounding and time bounding!). Now try to imagine the negotiations of this investor and a potential builder who prefer the agile approach saying: "Let's put aside now the details for the overall plan ? it will be changed so many times anyway; so let's focus only on the first 'time-box', i.e. building the apartments and the part of the main road to connect these apartments. We shall prepare documentation and the complete offer for that 'time-box', while the next 'time-box' we shall easily define and agree upon when we finish the first one. Thus we'll be able to start building much sooner since we don?t need to prepare documentation for the whole project, and we are going to save money because we won't have to update often that documentation for every change in the project"! Cheers! Perhaps even more illustrative example would be building the skyscraper: "Let's document and start building only the first ten storeys, and after that we'll see how we are going to proceed." In my opinion, that's the approach of producing soap TV series, and I would say that in such projects the matter of quality has probably the same relevance (that's why I mentioned earlier that agile is compromising the project quality).

anirbank2000
anirbank2000

I would tend to agree with the person who mentioned that Agile tends to compromise quality. If the design of the system is not done with a holistic view, the estimation will not work out. The reason being, there will be many blockades on the road, due to a faulty design and hence the actual hours spent will tend to be much higher than the estimated value. So Agile makes sense, as someone pointed out, when the Design is done with a holistic view and then the development is spread over iterations.

jcjr031064
jcjr031064

How do you convince your customer to take this iterative approach to estimating budget for the whole project. A customer might be hesitant to commit to that approach because he is afraid the cost might be more than what he is willing to spend. That if he had known the customer might not want to start the project in the first place.

RLN_Brian
RLN_Brian

Nice! This is a very pragmatic approach.

metalpro2005
metalpro2005

Have you ever seen a team of developers having to find solutions to real business problems in a short amount of time without having a (project) manager getting involved? These kind of situations often occur when 'the shit really hits the fan' and action is imminent. Have you ever noticed how effective and productive these teams are? And ever wondered why this is not the case in a fixed project environment? I think it has all to do with the clash of creativity and control. The agile methodology has been developed with these principles in mind. How do we maximize facilitating the development process, so the working environment of solving (business) problems is optimized? Often this methodology (like all methodologies) is misused as an excuse to start without knowing where to go. Or having no documentation at all. In the end optimal communication gives optimal results. Other methodologies are based on 19th century efficiency principles and have more focus on control and limiting creativity. Unfortunately most companies (and employee's mindsets I have to say) are also based on these ideas that's why there is a lot of sepsis around and success is limited. I do think the agile methodology gives the best bottom line and flexibility considering ALL costs and gains. But this heavily depends on the availability of LEADERSHIP (as opposed to management) in ALL roles concerned ! (and having an architect around with the 'current blueprint' that helps ! :) )

fparth
fparth

That's an odd comment in the article about PMI's "predictive" techniques. If you read the PMBOK, you'll see that PMI does not endorse any particular approach or technique. The PMBOK only says you have to plan, it doesn't give you a technique. In fact, PMI's IT department itself uses agile approaches.

GimbalOx
GimbalOx

It's been my impression that an agile approach to project management would lend the team towards an honest manner of approaching the problems for which a project would be addressed. There'd be less time spent in speculating, and more time spent in addressing the problems. I understand that the more traditional approaches such as SEI CMM would have their proponents, and I do not seek to argue about who's "more wrong" or "more right", any way it's layed out. Though my career has yet to progress to a point where I'd be working in a management position -- I will admit -- but, for what I've read, and for what I've picked up about the way things are done -- it's been my impression that a carefully applied agile method would make for a realistic and timely working approach to a set of problems. So, in final, I just wanted to make a note of appreciation for this article by Mr. Rick Freedman. The way I see it, the case is not closed with regards to agile methods. I think it's great to see such this substantial article about the same, then. Thank you.

azvonko
azvonko

After all arguments considered, I still find that the agile concept is a much more risky and hard-to-accept approach, and that's in fact a pretty much compromise to quality. Ok, one might say that it's the price of surviving in these days of too fast paced living and uncertainty, since usually there is no time enough to do the project "naturally" (more secure and more honestly) from the beginning to the end. Of course, whatever method used, in today's circumstances we are not able to completely avoid all risks of poorly defined requirements (as well as many other pitfalls on the project). But, in the case of agile, I think such risks are much more present than with some more predictive method. The bottom reason for that I find in the simple fact that agile ENCOURAGES lack of vision, poor planning and postponing of important decisions as far as possible. When we say "don?t estimate further than you can see" ["because it's not quite necessary to do it now, further estimation we'll do easily later when we get there"], that's often a dangerous and costly advice, especially when the customer really do not have a feasible vision of the future. Simply put, it's like we start the journey and say "Let's get the first flight, and after that we'll see where, how and if (!) we are going to proceed". Assuming that before starting we didn't have a more detailed vision of the path, time, costs, etc -- even if we had in mind a final destination -- after the first flight we could find out that we have to get back and take some other direction. And worse, without having precise enough vision of the final goal, after getting finally the destination of our "last-mind", we could find out that the price of wandering was simply too high. Therefore, it's obvious that we can much better minimize the risks and costs when we better define the path end-to-end in advance -- precisely as much as possible. Now, let's take into considerations also the usual truth that our goal might change in the mean time. That means that we have to maintain our project documentation (i.e. requirements), to keep it always up-to-date and, of course, to align our actions with the current version of it. For example, suppose that the sponsor suddenly made radical change in the ongoing project (due to his justified reasons); I still do not see that agile method would be in any advantage here over some more predictive methodology. A small advantage probably might be only some savings due to less frequent updating of the documentation (requirements). Shortly, I find that agile method leads to compromising of some substantial values in the project: developing of vision, more detailed planning, and finally the quality and cost of the entire project. So I think that the best way is (especially for "innovative" projects) to split the project in two major phases: a) Detailed enough system analysis which should yield enough information for design and all major decisions, from start to end; and b) Building of the chosen solution, while maintaining the actions aligned with necessary changes in the requirements.

fidlrjiffy
fidlrjiffy

The most important sentence in the entire article is above: "Estimation is a challenge in any project effort, but in these never-before-seen initiatives, it can seem impossible." I submit that by definition any and all projects are never-before-seen. Estimation is therefore like playing darts blindfolded. More importantly, the suggestions laid out apply to all projects and that agile, rather than being only for "innovative projects", can have broader applications particularly as an iterative framework.

keith.martin
keith.martin

Thank you for your article on agile versus predictive estimating. I have found agile estimating much easier and more accurate than estimates using a waterfall approach simply because estimating the iterations based on a good understanding of the work is much better than estimating the 1000 task project plan that is 85% based on guesses. Many of today's projects are innovative in nature. They are discovery efforts. They are not cookie-cutter repeats of previous projects that have clear roadmaps. There remains a segment of project management, however, that wants to remain mired in a mindset of projects of the 60-70's that lent themselves to predictive methodology. It is the classic case that if the only tool you have (a 1000 task waterfall project plan) is a hammer everything looks like a nail (a predictive estimation approach). I think the biggest take-away for young PM's from this article is the comment of not planning further than you can see. While it is important to have an overall game plan for the complete project, spending weeks flogging the 1000 task plan with the goal of achieving perfection is largely a waste of time. We always had a saying that the only thing we knew about the schedule we just created was that it was wrong. Build your boundaries of time and cost and then manage your deliveries near-term based on what you know. You will get there much quicker and at less cost than pursuing the Holy Grail of project management...the project plan and schedule that is all knowing.

Saurondor
Saurondor

Personally I believe your example of the skyscraper is not applicable. You confuse blueprints with the finished skyscraper. The blueprints is equivalent to the source code and the built skyscraper is the released code once the source is compiled. You must agree that the blueprint creation process was not linear. Surely a lot of ideas and models were drawn up prior to actually starting work on what would become the final project's blueprint. How many artistic drawings were made? How many models built? To say agile equivalent processes couldn't be applied or are not applied to buildings is just plain ridiculous. Skyscrapers are built a hundred times over in computer simulations before actual building begins. I like to quote a paragraph from your post which outlines a very clear difference between agile and predictive methodologies: "This leads me to an impression that agile is mostly preferred by the technically oriented IT people (what seems quite understandable), whilst the more predictive approach is better valued by the business oriented people, whose primary focus is on budgeting and contracting the project (also understandable)." This is the reason I prefer Agile over a more "predictive" approach. If I need to choose sides I prefer to stand with those who know what they're doing rather than with those who know what needs to be done, but are not sure how to. Or even if it's possible to do in the time frame they're arbitrarily setting. And I say they're arbitrary because the estimations made can and most surely do include an error. Like I mentioned in the my post, predictive methodologies assume that the prediction methods are 100% correct along the length of the project. And the longer or bigger the project the predictions become more complex and expensive to do. It also assumes the development process will go as anticipated, without any unexpected incidents or problems down the road. As von Moltke would say: No battle plan ever survives contact with the enemy.

tech
tech

I suggest you address the customer concern by assessing their desire to engage the risk of completion of the project to their satisfaction. Agile methods will give the customer a much earlier view of the product, allowing them to opt-out of the project and cut their losses than other traditional, seemingly more "cost known" methods. If they are more concerned about the acceptance of the end-product, they should opt for Agile and subject their risk model to iterative planning/costing; else, they can assume the risk of end-product acceptance, but know the final cost more definitively. Their choice!

wmjas.shaw
wmjas.shaw

Totally agree. What seems to get us into trouble is trying to bite off huge chunks instead of chewable ones. PMI does not recommend a particular approach, although the world seems to think that the PMBOK is synonymous with waterfall. I prefer shorter phases with decision milestones based on the successes achieved and the improved knowledge at the end of the phase. So, sort of Agile. You can start with an overall budget and timeline but you leave yourself the freedom to adjust course and possibly cancel the project at the end of each phase. In IT you need some fundamental structure: a good system architecture and good design can help avoid costly mistakes. Business Requirements are fundamental but it also seems next to impossible to nail down requirements: usually the customer has to see something working to know if it's what they want. I know the short phase approach is a hard sell to the executives who think they need a firm commitment to a completion date and a budget, but as long as we keep caving in to these unrealistic expectations we're going to continue to experience massive project failures. I've seen hundreds and hundreds of millions of dollars wasted on failed projects in government and banking and we all know who's really picking up the tab for these mistakes. Shorter phases with frequent milestones for review and decision making would reduce the risk significantly.

Tony Hopkinson
Tony Hopkinson

In business IT the the drive is good enough and cheap. Agile in the right project can make that trade off explicit or it can hide it. V (waterfall) nearly always hides it, and generally very successfully for a long time.... The quest to know everything you'll need to before start, including when you'll finsish and what impacts might occur during is at best naive. Business want deliverables, surprisingly they don't consider 4,500 UML diagrams a deliverable. That does not mean you have to buy into sheeple fashion that is agile though. Spiral an MoSCow work just fine as far as I'm concerned and if you are the sort of department that can only have one methodology no matter how stupid it is for a project, they'll work better than waterfall and glorified QAD.

wcoupe
wcoupe

Agile, in my opinion, is simply another method, that, properly applied can reduce the problems inherent in the waterfall methods. Improperly applied, it can bring it's own problems, there's no doubt about that. There is no magic bullet... all development is iterative, the question is where will the iteration, and refactoring take place? I find that utilization of the agile methodology, combined with some clear vision as to the end point, the total 'time-box' and the customer's budget constraints, provides a platform to deliver a product, comprised of the most important features (from the customer's perspective). Is it a perfect piece of software? Probably not, but it's the best fit for the constraints, delivered quickly and with the full understanding of what's being delivered by the customer as well.

esserlin
esserlin

The project is done in iterations - true. This doesn't mean you have buried your head in the sand regarding future iterations. You have to have your roadmap from the start and recognize that it will change over time. When it changes over time (as all requirements do), you will not have wasted time with requirements of future iterations. You must work with the business to determine the order of the deliverables and not arbitrarily decide to skip functionality that may seem difficult. I think you have misrepresented the agile concept just a bit.

wcoupe
wcoupe

It's also not a replacement for the customer articulating what they want up front. What it is, is a means of identifying what needs to be done first, with a focus on delivering value to the customer as early, and often, as is possible in the development process. To me, and I've been involved with just about every development methodology to come along over the past 20 plus years, the true brilliance in Agile is the total customer focus all along the path to completion. With each delivery of features the client is involved with UAT and can provide valuable feedback. Which, in and of itself, lends to a better product, for the customer, as they've been able to initiate course corrections all along the way. In addition, if the budget, or time, become constrained, once again the customer gets to decide what can be placed in the backlog (what features aren't 100% required) so the project fits withing their 'time box' or their budget. After delivering many projects over the past couple of years,using Agile, I can say that every single customer has been happy with the results. Given we have the same people, and the same customers, I have to give the Agile process, and the constant involvement of the customers, the credit for this result. In short, we're delivering, more of the features the customer actually wants, fully tested (as well as multiple cycles of regression testing on the early features) in less time, and on budget.

azvonko
azvonko

esserlin and apollyonbob, in a way you are both right: I am trying to talk abut the unreality of agile concept (or its very promise) in the reality (the everyday practice in the real world). Of course, from my experience and my perspective: Please let's better focus on requirements, that is - business requirements. Suppose you are working as a foreign IT consultant for the company where always the main (and very big) problem is incomplete and inconsistent definition of business requirements ? no matter how hard you try to help them in accomplishing that point. The lack of vision, even the lack of sight of the current system needs and possibilities, lead to hard negotiations, misunderstandings, often and hasty definitions and changes, very high technical and business risks, enormous project costs, etc. And at the end, IT (you inclusive) is always guilty for missing the target. Now appears agile, offering to business: "Well, actually you don't need to estimate now further than you can see, you'll do it easily later". Great! It sounds like "don't worry, be happy!" Therefore, it is not necessary to make NOW all those bothersome efforts in developing the vision of "big picture", in more detailed planning, in synchronization of wishes and system functionalities; it can be postponed for later, when the picture becomes more clear -- or when things become irremissible! That really reminds me to slogan "Do not miss to postpone today everything you can do tomorrow". Sorry, but I do not think that my example is so exotic. The ultimate truth is that for a good, CONSISTENT output (the final product) it's necessary to ensure the good enough, CONSISTENT input (business requirements). Finally, let me once again make use of a metaphor for comparison. I find agile as similar to approach of a young man who does not have any detailed vision of his wishes, ambitions, his farther future; he even does not feel the need for that vision, so he's just flying on the wind as a leaf, and makes decisions as issues appears. On the contrary, some more predictive method I'd compare to somebody who has pretty clear long term plans for his life, the final aim and the vision of the path to get there; though he knows very well that there will be many changes during his voyage, maybe even some radical changes of the final aim, he knows also that the best way to minimize surprises and maximize the outcome in that voyage is to maintain and follow his mindful vision.

apollyonbob
apollyonbob

I think he's talking about the agile reality, not the agile concept. If you're working on a large project, and trying to do it agile, my experience is that you're either going to spend a lot of time refactoring or you're going to be hacking code together. I get that the _concept_ of agile involves thinking about things that you're not actually working on at at the moment but the reality is, when you start working on something, the last thing you're going to do is think about something that's 2 iterations away, if it happens at all.

Saurondor
Saurondor

Your position is based greatly on the fact that all factors affecting a running project are external. More so the development team is not only flawless, but precognitive too! Let me explain. You mention that the business requirements are incomplete or maybe flawed and they will require tuning along the way. In that we are in agreement. You mention that the business requirements may change along the way. We are in agreement too. But you never mention possible flaws in transcribing business requirements into project specs. Somehow the design team gets all the business requirements into the best design in one shot without a single prototype. You never mention technological changes along the way or technological limitations found along the way and not foreseen in the initial planning. What if the business needs are well described, but the design and model are a bit flawed or not optimal. And based on this you go off to refine in great detail. If later you find that some of the key design decisions lead to bottle necks or said designs could be done better based on experience from working software (aka first prototypes or versions). You have just wasted a great deal of time on detailing things that will never come to be. One of the key benefits I see in Agile is addressing a blurry big picture rather than a crisp big picture. I don't think Agile ignores the big picture. It realizes that crisp detail costs money. Picture it as a snowflake. The outer crystalline pattern is dependent on the inner core's patter. If in real life you create a detail plan on building this snowflake. Then 80% along the way you realize that the core's crystalline structure will not allow you to build the desired pattern on the outside. Ups! Now you have to change a great deal of the 80% of the work done. Why? Because in classic top down models the design is made prior to any practical tests. These practical tests, usually called prototypes or something similar affect the design itself. We must consider that the project, its design and requirements are affected by the progress of the project itself. The development model you champion is equivalent to supercooling water (extracting enough energy for it to freeze without having it change state) and then perturbing it a bit in the hope that the shock waves will induce a crystallization exactly into the desired snowflake model. Except for the most trivial projects, I've never seen this happen in real life. An example that comes to mind right now is benchmarking. You can create a wonderfully detailed design of a system. Only to find performance bottle necks in the database. Or maybe your rich client application is using too much http traffic for the installed bandwidth. Early tests of key elements will help you save design time in unnecessary changes of dependent elements which will undoubtedly change if the depended on elements change. Another example is spending too much money in cooling the water that you have little money in actually crystallizing it into a snowflake. I've run into a few clients that rush to me with a well and thoroughly designed project (lots of docs, UMLs etc, the whole thing) and like 5 bucks left to implement it. They spent so much on designing it they don't have enough left to implement it. More so, the design is so vast it would take more money than initially budgeted to implement so many features. Half the features with one fifth the amount of time invested in design and specs would have resulted in a much more productive system than the documents and diagrams delivered. Which, to be honest, provide no business benefit (income). To sum it up. Agile promotes the idea that your initial design will not be the best nor the final. So don't work out every single detail before you start delivering workable products. Your design will evolve due to: badly defined business requirements which will be redefined better down the road, new business requirements that emerge or cease to exist and internal project issues which arise as the system is being developed. This last point is very hard if not impossible to see in great detail in the initial design phase. The fuzzier the initial design the quicker you can get down to address key design issues. The sooner you'll be able to say if and when you'll be able to deliver features based on these core elements. This leads to a crisper big picture that is delivered later down the road, but is based on real experience with the project and on real estimates. Rather than a crisp big pictured delivered early on, but based on hopes that all goes well. Which rarely happens.

Tony Hopkinson
Tony Hopkinson

The business goal is contant, clear well defined and terrifyingly simple. To make a profit. That's it, and I've got to say confusing the how with the what means a lot less what for those employing you. Your approach in any dynamic business environment is a well documented proven failure. Not because it's being done wrong, but because it can't be done. Denying that your environment has changed so you can successfully execute a project that has no contact with current reality, nvere mind the one that will exist on completion might get you a good mark in an exam, in business it gets you a kick in the arse and an overflight of the car park on your way out.

Tony Hopkinson
Tony Hopkinson

in any methodology should be approached with extreme caution. It's a guess based on a set of unproven assumptions at the time done by people with an imperfect understanding of then, now, soon and whenever. Agile (in fact any iterative lifecycle) greatest strength is that it limits business risk. If you are doing a lot of 'refactoring' (total misuse of the term by the way), you are either doing it wrong or you are using the wrong lifecycle.