It isn't your imagination; it really is getting more difficult to plan IT development. As systems become more complex, analyzing every detail in advance is beginning to stretch the limitations of even the best project managers.
But never underestimate what you can learn from the good times. When a project's going well, don't forget to ask the critical question: “Why are things running smoothly?” One likely (and partial) answer is that all team members and the customer are focused on their own contribution to the project and are clearly communicating with other team members.
When the project is going well, the team "self organizes." I'm not trying to put an end to the project management profession; yet, I have been investigating technical developments that, if they're successful, might greatly reduce the importance of our work.
Who’s in charge?
We're used to a structure that includes a project manager who plans and coordinates development activity. But in the real world, there are many natural systems that manage to do amazingly complex things without being managed. This is accomplished with numerous "team members" acting according to a small set of rules or principles. Such multiagent systems can support overall system behavior that's significantly more complex than the behavior of individual agents.
An ant colony is one example of a self-organizing system. They manage to feed, reproduce, and maintain their environment without being told what to do. Termites can even build elaborate, air conditioned, multistory nests using only a handful of simple principles. There is no "chief termite" to project manage the process.
Ants aren’t about to take the world of IT by storm, of course. But the ant's operating principles of self-organization are starting to move out of the lab and into the sphere of IT systems development.
Software behavior is already self-organizing
Originally, the basic unit of software was the entire program, which could contain large, hardwired jumps in control (e.g., like when using the dreaded GOTO statement). The programmer was completely in charge in terms of the software's behavior.
Then "structured programming" worked with smaller chunks of code containing loops and subroutines. The subroutines gained control of the behavior only when called. Next, object-oriented programming (OOP) localized code segments and also variables. Objects were passive and only gained control when sent a message.
When using software agents, each object has its own thread of control and its own personal goal. This localizes code, data, and invocation and is pretty much the level at which ant colonies operate.
The next advance in software now under development gives agents high-level goals and the ability to adapt their own rules according to their experience of what works.
Dynamic Web pages are already self-organizing in a primitive way. The content can be organized on the fly, and in accordance with simple rules derived from your personalization profile. XML is intended to extend this facility so that data carry a repertoire of possible ways in which they can be allowed to "behave."
More on self-organizing software
These Web sites offer more information on self-organizing software:
Web services also represent a form of self-organizing system. The overall function is performed by a collection of individual services that compete (via resource discovery) to be the most appropriate on any given occasion. Big corporations such as Xerox and IBM are actively working on new methods of routing messages efficiently in peer-to-peer networks, using self-organizing techniques.
Is the future in self-organizing applications?
When cool-headed project managers find that software system requirements are too numerous, changeable, and complex to specify feasibly, self-organizing systems may provide the only way to cope.
Self-organizing applications (SOAs) are being designed to evolve in response to dynamically changing requirement specifications. SOAs are typically based on several interacting software components that can act autonomously and in collaboration with each other and with no central organizing entity.
Rather than put in place any recognizable project structure, the developer of the future may simply select the required agents for the final application and let them organize themselves to provide the required functionality. This may result in enormous benefits in terms of adaptability, robustness, and much reduced development costs.
Until that day dawns, the best projects are still the ones in which the rules and communication channels are clear enough for the team members to self-organize.