Software Development

Consultants: It's not the theory, it's the execution

Project success has less to do with the adoption of a specific methodology or theory and more with how well the people involved can adapt the procedures to new challenges. Chip Camden offers more insight into this observation.

 Of the 18 IT consulting maxims I listed in my previous post, number three -- It's not the theory, it's the execution -- is the only one that I haven't written about before, which is odd because it's one of my favorite mantras. Let's expand on this observation.

Managers often fall victim to the notion that "if we adopt a specific methodology, we'll fix everything"; this mindset can infect consultants and developers as well. We're all tempted by the promise of not having to think about a certain subset of the actions we must perform; if we just follow the prescribed procedure, we'll be all right.

You don't want to have to reinvent all of your processes on every project, so you should standardize procedures to help avoid missing things. But nobody has perfected a methodology for a nontrivial activity yet, so you always need to consider when to break the rules -- or at least bend them a little.

As an example, let's consider the Agile vs. Waterfall controversy. I'm a big fan of Agile principles (I even signed the Agile Manifesto early enough to get my name in the left column), but just "doing Agile" doesn't necessarily fix anything. Here are ways that Agile can go wrong:

  • You can fail to plan far enough ahead.
  • Scope creep can eat all your resources.
  • The product's design can end up inconsistent and not well thought out.
  • You may never get to "done."

Agile doesn't cause any of these scenarios -- people who do Agile badly cause these scenarios.

Now let's consider the all-too-familiar drawbacks of the Waterfall approach:

  • You'll probably fail to account for some important requirements.
  • When you add or modify requirements during a later phase, it requires massive rescheduling.
  • The initial design is almost never right, and you won't know that until implementation.
  • You have to be omniscient to be able to budget the required time and money.

Despite these drawbacks, it's possible to use the Waterfall approach and still be successful. How? By taking a few hints from Agile processes.

  • Change the requirements or design whenever necessary -- after verifying with all stakeholders. Build extra time into the schedule for these mini-iterations.
  • Keep an open mind about the phases. Don't be so stuck on defining the phases that you find yourself on a death march through them.
  • Start prototyping during the research phase rather than waiting until the design is finished.
  • Organize the project so the less critical features are implemented last; this allows you to omit the features if you run out of time.

To be successful with Agile, you can't blindly follow its principles either. You should keep in mind the following:

  • Some projects or parts thereof benefit from thorough planning and design in advance.
  • Sometimes you need to say "no" to user requests. (Unfortunately, not all user stories have happy endings.)
  • Sometimes you have to place limits on a project's schedule.

No methodology or theory is a silver bullet. You can use any system to help you organize your processes -- though arguably some are better than others depending on what you're trying to accomplish. The lessons we learn from the failures and successes of each approach are far more valuable than adopting one or the other. Project success depends much more heavily on the people involved, and their ability to adapt the procedures as needed to meet the challenges they face.

The same principle applies to many domains. I've seen object-oriented code that clearly represented components of an application, and all developers have seen object-oriented code that obfuscated what was going on in order to satisfy a purist's notion of what object-oriented programming should be. Ruby (which is one of my favorite languages because it is so expressive) can very easily be monkey patched into an indecipherable nightmare, while I've seen PHP (a language that encourages badly organized code to the point that I've often said it must stand for Page Hacked to Pasta) that was so clear and well-organized it would make you weep tears of joy. 

The theory is not as important as the way you put it into practice.

Get weekly consulting tips in your inbox TechRepublic's IT Consultant newsletter, delivered each Monday, offers tips on how to attract customers, build your business, and increase your technical skills in order to get the job done. Automatically sign up today!

About

Chip Camden has been programming since 1978, and he's still not done. An independent consultant since 1991, Chip specializes in software development tools, languages, and migration to new technology. Besides writing for TechRepublic's IT Consultant b...

26 comments
tburkett
tburkett

I really enjoyed your column. It sounds like we need to remember to insert the ART form into our project methodology. As an old CHIT manager, we use this strategy every time we needed to get a project done. (A - Always begin promptly R - Reject all negative thought T - Take action now and adjust when necessary) Let the CITS fall!

caryl.forrest
caryl.forrest

Ability to change direction rapidly is a major point of differentation between large corporate/bureaucratic organisations and small, enthusiastic software shops. There's a mindset in larger organisations that says if we've done the right formal methodology - and it's not just waterfall or agile, it's also the standard project management methodologies as well - the software development project will "work." This absolves all parties from responsibility as they've done things right. Never mind that it's not the right thing. Small, adroit enthusiastic software firms are not part of this large, lumbering juggernaut; they see something isn't going to fly, have a brainstorm and take off in a better direction. We need to move from the methodology/process is everything mindset and introduce a novel concept - common sense.

azvonko
azvonko

Generally, yes, I agree, but ... Let's be a little bit more careful in expression. "The theory is not as important as the way you put it into practice." Doesn't this still sound a little bit biased, but now in the opposite direction (unintentionally, of course)? I mean, it shouldn't encourage the undervaluation of the theory. From my experience -- and not only in the domain of IT project management and consulting, but in the life generally -- I find that much more often the bigger issue in problem approaching, decision making and communication is the ignorance of theoretical foundations than the overvaluation of the theory.

PMPsicle
PMPsicle

With your suggestions for Waterfall ... "Change the requirements or design whenever necessary ? after verifying with all stakeholders. Build extra time into the schedule for these mini-iterations." That's what change control is for ... and the Management Contingency Reserve. Even if most of us use the Contingency reserve. Us bad ... part of the reason we have such a high project failure rate! Waterfall never claimed that change wouldn't occur. Just that you need to get things right before you go on to the next stage ... and you should never need to go back more than one stage (if you're in coding and discover you did the requirements wrong ... then your process is bad 'cause detail design should have found the problem!). It's 1:30 in the morning ... it's either nitpick or sleep ... so I'm gonna nitpick! Glen Ford, PMP http://www.trainingnow.ca

avidtrober
avidtrober

Agreed. My shop's done methodology consulting 10 years, and swimming against the current of mainstream hype. The most effective methodology is transparent to focusing on the actual problem at-hand. In our field, we have to overcome the hype, e.g. Agile is nothing, absolutely nothing new. Yet there are so many 'deer in the headlights' stares from the uninformed. I with there was a totally different way to go about methodology training and tools. But, no matter what's done, there are those who apply a 'recipe' or 'formula' somewhat like a playbook. IMHO, those are the ones who shouldn't be calling the shots on projects, yet many are.

Sterling chip Camden
Sterling chip Camden

The methodology of love espoused by many religions certainly has some odd applications in practice. The theory of tolerance among the non-religious can sometimes claim an unreasonable exemption against religion. Conversely, both religious people and atheists can be very reasonable people, depending on how they practice what they preach. The same goes for politics. Some conservatives actually do have a heart, and some liberals do have a brain. Both come to light only when they bend the dictates of their political ideals to match the needs of the real world.

Sterling chip Camden
Sterling chip Camden

The T in your acronym corresponds with the thrust of my article. The A and R just provided me with some thoughts for further exploration.

Sterling chip Camden
Sterling chip Camden

How you execute includes responding to contingencies that don't fit your theoretical model. Size also affects another important factor: survival. A small company often cannot afford to be wrong for very long. A big one really can't either, but it's easy for people in a big company to feel that the sheer size of the organization will keep it afloat. Q: "What could one project failure do to a company this size?" A: "For want of a nail..."

PMPsicle
PMPsicle

is that both "big" corporations and "small" suffer from the same problems. Although culture does play a part. I've been in both situations. The methodologies (in general) exist because of the problems they are intended to correct. (And of course, because somebody figured they could make a mess load of money from it. Although that's more true of agile than waterfall.) Waterfall was created because these small, adroit software companies were taking off in a million different directions and producing crap. Bluntly put they were changing direction so frequently that the programs were being screwed. When you spend $500,000 on a single program (in 1960s dollars) controlling a $10,000,000 piece of equipment (1960s remember) it sort of makes sense to be able to prove it works before committing the $10,000,000 and to be able to reuse it afterwards. That's why discipline usually is defined. Agile was created because a)waterfall required a level of discipline which was higher than most companies were willing to spend to support, b) it required an ability (conceptualization) which many people are not capable of, c) it requires planning which is a capability which modern corporations seem to be incapable of, and d) most of all, it was a way to make money. (For those of you who are wondering -- yes, I did limit my reasons to only those that were true ... so if your favourite is missing there may be a reason). Are some companies unwilling to admit they are wrong ?... yup. Are some companies willing to admit they are wrong ?... yup. Does it have anything to do with size ?... nope. Does it have anything to do with culture (specifically the culture of fear)? ... absolutely. I've seen big companies where fear is the common thread of their culture ... and I've seen small companies where it's a keystone. I've seen big companies which don't seem to include it in their style and small companies where risk is part of the game. If we are or ever are to be professionals we need to a)know the disciplines necessary to our success b) know the appropriate methodologies which inculcate those disciplines and c) most of all, know when they apply and when they don't apply. Glen Ford, PMP http://www.trainingnow.ca

Sterling chip Camden
Sterling chip Camden

Yes, lacking a good theoretical base from which to design things often leads to going down blind alleys or making poor choices that could have been avoided. I'm not advocating cowboy coding or its equivalent in other realms. But I often encounter people who have "got religion" over some theory or other, and who just as blindly apply it to everything. You need a good theoretical base, but you also need to apply the theories wisely -- and know when they don't apply.

Sterling chip Camden
Sterling chip Camden

As one of my colleagues used to say, "should and does ain't the same thing." the more complex the system, especially the more that it requires development of new systems or technologies, the more the chances of adequately discovering all the requirements before the end of the design phase approaches zero. You may say that the design phase wasn't really "done" in that case, but I'd counter that such a design is sometimes never "done".

PMPsicle
PMPsicle

Maybe someone will pay me lots of money! :D At least until they try it. There's an old saying that the key to success with a rule is in knowing when and how to bend/break it! Personally, I find it hilarious to see people try to drive screws with a hammer. And conversely a nail with a screwdriver. What I like to do is point out to the Agile-is-perfect folks is that their flagship project was cancelled after 10 years of never being implemented. Boy do they get upset! I'm mean.... ]:) (Unfortunately, it doesn't work with the Waterfallers ... most of them are too old to get upset anymore and their early projects all succeeded... comparatively.) Glen Ford, PMP http://www.trainingnow.ca P.S. All of which is to say, that you are correct. Yes, there are reasons to use one tool over another in a particular situation. But never forget ... it's just one more tool in your toolbox. It won't work all the time. Nor is it the best solution -- all the time. P.P.S. I'm old enough to remember DFDs and the argument between the Yourdonites and the Gee'n'Esses. Personally, I used Yourdon when working with the users because it was so informal and G&S when doing a formal document (because it was easy to make look neat ... as in not sloppy). Drove people nuts then too! Some things never change. ;\

Sterling chip Camden
Sterling chip Camden

I think the degree to which someone treats a methodology as gospel is inversely proportional to their real understanding of what they're doing. They need the methodology because otherwise they're out of control. As people become more confident in their own understanding of their processes, then they use a methodology more for advice and reflection rather than as a set of rules.

Sterling chip Camden
Sterling chip Camden

You're right that the size of the company doesn't exclusively determine its culture. But the larger the company the more likely that one person can't call BS and change the system.

PMPsicle
PMPsicle

The first being ... that's why you need to have multiple tools in your toolbox. Hammers are great until you try to screw a screw in with one. The second is ... that's what testing is all about. The one advantage that waterfall had over agile was the frequent testing at each stage. The advantage that agile has is that with agile, the testing doesn't require the user to have the ability to conceptualize. Glen

santeewelding
santeewelding

You are sounding as though you scratch at the door.

PMPsicle
PMPsicle

I think your next comment {"I think you've rephrased my point") is another key point. Some companies live in fear ... fear of being fired if they're wrong. Some companies never question past decisions. Those companies will ride a failed project into the ground. When that fear of failing is replaced by a greater fear then people will be willing to "call BS" (another great phrase Chip!). Any PRIVATE (closely held) company (and yes, that's usually small companies) has at least one person with a much greater fear -- the fear of the company failing. It's easy to dismiss the fear of being wrong when you're worried about the survival of the company you own and that supports your lifestyle. When did Ford cancel the agile mess? When it was losing big money and the Ford in charge put down his foot and demanded cost reductions. The fear of admitting failure was replaced by a greater fear of not cutting costs. The advantage that small companies have is that the boss is usually close enough that they can "call BS" and then hide it if they aren't the owner. If you want proof that size isn't a key factor try killing a project that the CEO has to report to the board as his responsibility! Glen Ford

Sterling chip Camden
Sterling chip Camden

TDD helps with that if you develop a set of tests, show them to the user, and get them to say "OK, if it passes all those tests then it's what we're asking for." Then if it turns out that they wanted something else, it's on them to approve additional funds for the change request.

PMPsicle
PMPsicle

Absolutely true ... The only problem is getting the user to either conceptualize the solution (hey it does the job) or to accept if it doesn't look the way they expected. Which, of course, has been the traditional bug-a-boo all along. If the user can't touch and feel the solution, how can they approve the design. In architecture the answer is the architectural model. Which is an expensive answer to the problem -- but one which is used when the cost (or the aspirin costs of dealing with the client) justifies it. In information systems it's very difficult to provide a useable model especially when the design involves process change. (??? is it test driven design or design driven testing ???). Glen

Sterling chip Camden
Sterling chip Camden

Test-Driven Design can work with either waterfall or agile -- write the tests for the behavior you're implementing before you implement it. It does save a lot of grief, and often reveals inadequacies in the design early on.

santeewelding
santeewelding

Meaning the original. "Here" meaning your translation (above) into religious and political terms.

Editor's Picks