IT Employment

The roots of agile project management

Here's a brief history of agile project management. By brushing up on these fundamental concepts, you'll gain insight into the challenges and problems that agile techniques are designed to resolve.

My recent agile project management columns focus on practical elements, such as project sizing, planning, and estimating. Now I'll take a step back and look at some of the theory behind the agile project management movement.

Most IT project managers and software developers are familiar with the Agile Manifesto, the foundation document of the agile movement, but most IT pros are not aware of the philosophical underpinnings of this movement. The ideas that support agile development and project management didn't spontaneously spring out of the minds of the signatories to the Agile Manifesto; the ideas are based on a history of academic studies and real-world experience. Knowing these fundamental concepts will add depth and context to any discussion of agile methods.

Let's start by acknowledging that there are accepted issues with standard project management methods. The famous Standish Group CHAOS Studies demonstrate that many IT projects fail to fulfill schedule and cost forecasts, and often fail to deliver the benefits predicted. These issues have been confirmed by various organizations, including the Department of Defense (DoD). The DoD noted that, of the $35.7 billion spent by the organization in 1995 for software, only 2 percent of the software was usable as delivered. The DoD found that 75 percent of the software developed was either never used or was cancelled prior to delivery.

Other academic research challenged common IT development and project management methods. In 1998, Harvard Business School academics Robert D. Austin and Richard L. Nolan studied large software projects. Their study, which questioned many of the fundamental ideas of IT development and project management, produced these key findings:

  • "The first flawed assumption is that it is possible to plan such a large project.
  • The second flawed assumption is that it is possible to protect against late changes.
  • The third flawed assumption is that it even makes sense to lock in big projects early."

Watts Humphrey, a respected IBM researcher, followed this study with a paper outlining his Requirements Uncertainty Principle, which asserts that:

"for a new software system, the requirements will not be completely known until after the users have used it."

Hadar Ziv of the University of California followed soon afterwards with his Uncertainty Principle in Software Engineering, which states:

"Uncertainty is inherent and inevitable in software development processes and products."

The connection between these ideas and the underlying concepts of agile project management should be clear. If users can't foretell what they'll want until they see it, if predicting and planning substantial IT projects is not possible, and if protecting projects against changes that arise during the development process is impractical, the ideas behind existing "waterfall" methods are clearly flawed, and an incremental, prototype-based methodology could offer substantial benefits.

The rise of the Internet ushered in a wildly innovative and experimental atmosphere in IT. Alan MacCormack, assistant professor at Harvard Business School, and two of his colleagues surveyed the software development methods of innovative Internet companies. In 2001, MacCormack's influential article Evolutionary Model of Software Development Methods outlined a history of IT development techniques, which include these models:

  • Waterfall: follows a sequential process and maintains a document trail.
  • Rapid prototyping: creates a disposable prototype which is exposed to the sponsor to establish customer preferences.
  • Spiral: delivers a series of prototypes that incrementally incorporate user requirements.
  • Incremental or staged delivery: delivers a system to customer in chunks of functional programs that are integrated incrementally to create a complete system.
  • Evolutionary delivery: offers an iterative approach in which customers test an actual version of the software.

Simply recognizing problems with existing methods does not solve them. In MacCormack's article about Internet companies, he recommended a set of practices that he believed could begin to replace the traditional methods. These simple precepts (which will be familiar to anyone who has researched agile project management ideas) have been cited as launching the movement towards agile techniques:

  • Early release of evolving design and code,
  • Daily build of code and fast turnaround on changes,
  • Deeply skilled teams.

The Agile Manifesto was the culmination of these new theories and approaches. Written in 2001 by a group of advocates of iterative and incremental development methods, this simple statement is the foundation document of the agile movement, and sets forth the underlying philosophical concepts of agile project management. The signatories include many of the founding fathers of some well-known agile methodologies. Signer Kent Beck went on to develop Extreme Programming; Alistair Cockburn became the developer of Crystal Methods and author of influential works on agile development; and Jim Highsmith has translated agile software concepts into an Agile Project Management methodology.

Understanding the academic and experiential background of agile methods better positions us to have a persuasive conversation with our sponsors and clients, and enriches our insight into the challenges and problems that agile techniques are designed to resolve. Agile methods are not simply "sanctioned hacking" as they are sometimes caricatured, but are based on solid research, demonstrating that these incremental, iterative, prototype-based methods solve problems and offer real benefits.

Get weekly PM tips in your inbox TechRepublic's IT Project Management newsletter, delivered on Wednesday, offers tips to help keep project managers and their teams on track. Automatically sign up today!

About

Rick Freedman is the author of three books on IT consulting, including "The IT Consultant." Rick is an independent consultant and trainer, working, through his company Consulting Strategies Inc., to help agile teams and organizations understand agile...

10 comments
weising
weising

during my past 12 years of experience: no matter how early we encounter the users to the projects, they would still fail if we didn't have highly skilled and experienced developers. The "skilled" here I am talking about is the skill to design the software flexible enough to be modified in a very short period; and "experienced" is the experiences to work with those clients who ALWAYS hand out useless ideas and material and create the mess during the development. In Taiwan, most of the clients tend to consider themselves highly intelligent but in fact knows nothing but creating bigger confusion on the business model. If we don't have the experienced developers who are capable to shut them up, no project can be delivered successfully.

pradeepbhanot
pradeepbhanot

Rick, excellent posting. You make some really insightful points on the nature of requirements and how they invariably change and the customer never really knows what he or she asked for until they get hands-on. Agile methods are essential for projects that follow a critical path or have a high degree of human interaction. A hybrid approach, melding traditional project management methods with an Agile approach still have value in providing the context for the overall project.

paul.simmons
paul.simmons

The points in the article are all valid. Ideally one wants to design the back end data structure and programs to meet the needs. A great program and data design can quickly turn into a bad design when functionality changes and these are patched in fast. I believe that a lot of large spec changes are preventable. The problem is finding good designers to pull out of the users the best possible design the first time. The notion that users know how to spec a system is just crazy. If they knew what they were doing they would not change their minds. We wind up paying enormous amounts of money to provide educational user experience in how to fail at designing systems.

minstrelmike
minstrelmike

I suspect only part of the problem is that users don't know what they want until they see it. We do a lot of upfront analysis to know what questions need to be answered and that drives the data. As far as the form layouts, those are 1) easily changed and 2) can be mocked up quickly. To me, the real reason specs change is because reality changes. If you are trying to meet a market need or a future need, it will probably have changed by the time you get there, so you need to be able to change things almost as fast as reality changes. That's the real use of agile and also signals its lack of acceptance--only businesses that wish to meet reality on its terms find benefit of agile. For everyone else, it seems like you're asking too many questions too often and are being insolent to the wise managers who initiated the long death march project in the first place.

Tony Hopkinson
Tony Hopkinson

Far too often that gets interpreted as not bother with the change and then give them something they don't want. If (who are we kidding, When) functionality changes durng the life time of a project. If you are Agile, you present a choice. You can keep going this way and muddle through till the next version, you can encompass the change instead of these, you can extend the deadline and the cost, you can even bodge at a cost to phase N + 1. If that isn't what is happening then you are not agile, you are simply leaping about putting fires out. Damn Sorry bin caught fire Right where was I. Agile project management requires agile business management.

addicted2speed
addicted2speed

You're right about End Users not knowing how to spec a system... however I would add that Developers have no idea either... until the End Result is agreed-upon... and the Business unit is well-informed prior to their decisions... meaning we need to do a little bit of education and guidance through the Business Requirements phase, and not simply to document the request word-for-word. For the sake of abstract simplicity, I'll use mathematics. Say you have two systems: We'll call them "3" and "4". The Business unit says they want to combine systems 3 and 4. So you ask the End User whether 3 + 4 = 7 is what they want, they say, "Yes". Ah. But if you stop here, then you're just asking for trouble. The End User may not know about 3 x 4 = 12, which would be a far-more productive solution. They come back to you later and say, "I want 3 x 4 = 12", and from their standpoint, it is no different than the original 3 + 4 = 7 request... but this now results in a Spec change on the IT side... simply because we did not ask the question: You're asking for 3 + 4 = 7... would 3 x 4 = 12 be a better solution for you?

asimakis
asimakis

I have to agree, one of the benefits I have seen with agile is to aid with managing changing requirements. Specifically when business realities change and what we wanted 6 months or a year ago when we started documented requirements has now become obsolete or no longer necessary based on new economic landscape.

Curtis R. Unruh
Curtis R. Unruh

It's all waterfall or it's doomed to failure. You can call it anything you want, but you are still planning, gathering requirements, doing analysis, desgin, development, testing, deployment. You may do it one time or six, you may spend more time on one than another, but you're still doing it all. Where IT people add value is the applying valid engineering princles to software and system engineering that allow us to replicate success. Call it whatever you like.

Tony Hopkinson
Tony Hopkinson

Waterfall is not the steps in the process, it's that each one must be complete (effectively) before the next is started! Try to keep up.

Editor's Picks