Discussion on:

10 classic mistakes that plague software development projects

... or cure world hunger.

Many projects are just too big, covering too much scope. In many cases it is better to move forward incrementally - even within a larger scope of work. Concentrate on getting a well-defined, easier-to-manage set of tasks delivered and bedding down before moving forward with the next. In many cases, massive "big-bang" data migrations fail due to their complexity or end-users simply cannot absorb enough simultaneous change to support successful project completion.

Note: I don't even necessarily mean use a Rapid delivery methodology here ... just define a scope that is plausible and deliver it. Success breeds success.

It's called global warming
But this is indeed a major issue with some projects, and they tend to grow accordingly too. If you don't have a project leader who has a strong backbone then, you get a never finishing project until it's trashed.

Justin, I really like this post. From my perspective the biggest hurdle out of the ten is constantly facing the 'Wall of Process'. This is especially true in larger ITs when we are working in an Agile manner and their processess have been established to 'control' everything. I like to think of this as the Productivity to Risk model. The more an IT tries to control risk the less productive they become. Of course risk mitigation is warranted; however, the problem is that many ITs only have one risk management process which is set to deal with the highest degree of risk making everything terribly inefficient.

Your #6 hit then nail on the head for me. I just left a Software development project after having been told I was not a programmer, and locked out of the loop. Even though my project will be distributed. They will have to rewrite a lot of it because I was not privy to their discusssions. And, therefore, not compliant to thier standards. Without communication there is no collaboration, no synergy and the final product will never be as good as it possibly could. I remember videos of the early Atari crew in all their capes blowing off steam. That comraderie made Atari a leader for quite some time.

Justin, apprecieate the post. My views are related to the "Wall of Process" or managing the changes of requirements that appeared to be poorly documented by the "business" (even if detected by risk-analysis, the business claims to be able to provide the details if requested), the change of persons (business' POC changed three times, got overcommmitted, do not remember, unable to understand the questions of the contractor, ...). So I decided to build upon the US DoD practices of "Request for ..." process. Here again, we had to define the process ... business rejected the procedure (we are perfect syndrom), Contract agrees but once needed, Legal and I stated that the company is mandate to write down all their claims (Conformity, Price, Delay) if it impacts the contract. So, company and customer share the "wall process" .. I do the initial draft and they complement my questions.

To make it short, this is about documenting changes of personnal in case of claims. In Public Procurement, there are penalties for late delivery including non-conformity to specifications.

I work in a University, not in an enterprise, but nevertheless I recognize some situation I went (and I am still going) through...

1. I also call this "the single shovel syndrome." The name refers to a commonly made remark about road works: "When there is some road maintenance work, you see ten people, but only one is digging." Although, personally, I do not believe that this is always true, it could actually happen if your work plan calls for a hole to be dug and there is only one shovel: it does not matter how many people you put a work, only one will be able to help. This is actually a bit different from the "pregnant woman" effect: pregnancy is a strictly sequential task and it cannot be split in parallel, hole digging can be split, but you must provide enough resources (shovels) to get advantage from parallelism.

3 & 4 (about time estimations). When we submit projects to the EU for funding we are asked for a timeline that extends for 2 or 3 years. We should specify how long that workpackage about development of a cloud 3D sharing platform (or whatever) will last. My first reaction is "How in the (whatever) am I supposed to know?!?" It depends on many variables that, since I do not have a crystal ball, I am unable to predict... It depends on the ability of the people that I will hire to write the software, it depends on the outcome of other WPs, we could have problems in getting SW and/or HW that will delay the development. In these case I (like anyone else, as far as I know) just throw some numbers that can look reasonable. BTW, I decided to read Brooks' book to actually get some hints about stuff like this.

9 & 10. I summarize this in the motto "people are not computers." With computers you can expect them working since the time you plug them and you can use them with lots of different tasks, but people do not work in this way; they are more slow, more "low pass" (to use an electrical expression) and they have lots of overhead when switching tasks.

I see 9 especially with some people from industry that come to me with a request like this: "Oh, we need some young graduate, it must be an expert on (a very obscure piece a SW that only they use), since we must deliver a product in two months and we are searching for someone who will be productive since day 1." Typically I am kind enough and tell them that I will ask around for someone, but in my mind I think "You just got your times wrong," ask to a couple of people (just to say to myself that I tried) then remove their request from my mind.

I see 10 with many coworkers of mine. They get a student to work on some project (that does not interest them, but it is founded) and they would like that the student worked also on a different project (that interests the professor more, but it is not directly founded). The overall result is that, if correctly handled, the student does a good work only in one project, otherwise does a bad work in both projects.

Premature optimisation, turd polishing, dashboard mania, reality being incorrect as it doesn't match the plan, scope creep, the agile waterfall development lifecycle, technical debt, releasing prototypes, ad nauseum

In short a failure to realise that this is IT and change is a given.

Tech companies seem to have a special skill at failure to communicate. In my work as a software developer and tech writer I have observed several difficulties:
- Failure to keep projects secret from competitors by failing to instruct developers on secrecy. In 1976, as a DEC customer, I encountered one of DEC's dev gurus at a party and asked him about the possibility of a 32-bit version of the PDP-11. I waved my hands describing two possible architectures, and may have mentioned Multics-style virtual memory. He said, "I know too much to say anything, but basically you are correct." Oops! If I knew about it, then everyone on three planets knew as well. My friend and the rest of DEC's VAX team should have been instructed to say, "Fascinating," and nothing more.
- Failure to give secret knowledge to co-worker teams. Too many times I have seen the doc group get a request for documentation for a product just six weeks before its shipping date. Frequently the tech writer finds blocking bugs as she explores the product, and of course she gets blamed for the slippage.
- Failure of management to listen to "the trenches." This is your classic Dilbert storyline, where management punishes messengers who try to get help for problems that the developers are encountering. An infrastructure to hide bad news from management develops.
- Operation of hidden agendas. The folks in the trenches never find out until much later that their much-touted project was deliberately doomed for some nefarious purpose. The company's stock zooms and crashes on schedule, management's confederates make a bundle, and any whistle blowers are accused of paranoia. The best employees leave in disgust, management "restructures" the remainder, but cannot attract creativity while forcing it away, and bankruptcy follows.

It's not all gloom. Occasionally things go well. Lessons from Moore, Brooks and Ershov are applied, excellent software and hardware go out the door, operation in the field is straightforward, IT crews are slightly happy, and customers are pleased.

Very good post, very well thought. My favorite is #9 - nobody ever thinks about a person added to a project will need to be "trained" on that project, and also who will do it. The worst is interns, not that I don't like interns, but I slows you down when you have to teach and do at the same time.

Point #1, about not throwing large numbers of workers into the pot, has another aspect that's often ignored. The more people working on a project, the less each is responsible for its success. This works against a sense of individual responsibility, and the resulting pride "in a job well-done". Workers should be added only as they are actually needed.

Not having a good induction plan (in the coding team) to introduce the new comer to the project e.g coding standards, code architecture and special components used in the software development process. This might be a resource consuming activity in short-term but it pays out in a long run

I'm told Microsoft does this, and it is surely not alone. It can be seen in the gradual deterioration of such products as Word and Visio. There should be no more than one or two people who make decisions about features and interface.

A common hazard is taking too long a view, plan too far ahead and show no intermediate results to the end users. The result is a project that gradually looses credibility and support. The end users generally react negatively when (finally) results are shown to them. In a worst case scenario the goals to be achieved may have changed over time, rendering the project results partly or totally useless.
Stick to incremental result delivery with a max event horizon of 4 to 6 months!

I seems to be especially the case working for the public sector, but the ever moving target, and/or flip-flopping of or specification of conflicting requirements has been the biggest bug-bear in over-running -projects I've been involved in. How many progress meetings have I endured the heads of two user departments bickering about their requirement being more important than the others, or the "that's not quite what we meant when we specified this".... "oh hang on yes we did".... "erm did we?"... "oh that was before we has the re-org, but now..."

This is what leads to the process wall, as the bickering turns to mud-slinging in which the only place it sticks is on the development team.

That's what I called number 1. It applies to all cases of project management. 20 years ago I too was against (leery) of multitasking, it doesn't work!

Study SAIC and CITYTIME to learn everything about a totally botched project. And do not EVER send software development to India.

You seem to think folks in US / Europe are computers and not human beings... not even a single bug/error has ever crept in .right!!.come on...every one makes mistakes..i presume u must hv had bad experience..but that doesnt mean labelling entire country as bad..this is RACISM at its best !! Thank god we dont much people like you running IT.

It is a nationality. Now, the OP may be a nationalist, or he may be expressing his experience. I've had some professional fratsority brethren who were descended from people who had lived in India and, other than confusion over how to pronounce their names properly which you get between any 2 dialects or languages, they were OK folks, no problem with a sister marrying, retaining one of them to be my doctor once they got their MDs, to develop the weather forecast before my flight, etc.

OTOH, I've had neighbors who were from India in the USA on H-1B visas, supposedly data-base experts, and they were totally clueless, based on attempts to make professional small-talk, so I can see that some such might have bad reputations when it comes to the quality of their work. And quite a few have trouble understanding and expressing themselves in colloquial American English.

If anything, the one published criticism I've seen of SOME people on H-1Bs from India and recent immigrants from India, is that their educations relied too much on rote memorization. They may be able to correctly cite various aspects of programming standards, but seem to have trouble applying software development principles in practical settings. Former cross-border bodyshopper Vivek Wadhwa compared most STEM workers educated in India to having "static" knowledge, versus more "dynamic" application of concepts -- creative, innovative -- that is more common among US and UK educated STEM workers. Attending US or UK universities helps.

It's only an issue because so many H-1B and L-1 (and equivalent) visas are issued to people from India, which does displace and drive down compensation for and disrupt careers of bright, capable, indusrious US and UK STEM workers. If reasonable numbers of such visas were give out, it would never be an issue; it wouldn't matter whether 99% of them were excellent or 99% of them totally incompetent. But people who believe they (and their friends and/or relatives) have been displaced by people who they see as less excellent, or whose work negatively affects the success of projects of which they are a part is irritating, so they lash out... with a broader and broader brush as the experience is repeated.

If a PM or team manager can't say no, scope creep comes into play too often. You have to know when to say "yes" and when to say "no, that will wait for the next version or update."

Scope creep/creeping requirements are one of my bug bears. PMs need to learn to say no in the right way to directors and important customers or the integrity of the whole development process is compromised. This will eventually affect the quality of the product and later the companys image in the eyes of the customer. A PM or development manager that knows how to get this concept across to directors in an affective way is worth his weight in gold.

Great post, Justin! I agree that all of these are major factors in why IT projects fail and why IT is often not viewed as a valued business partner and more of a necessary evil in the business. Here's blog post with some additional thoughts on IT project failure centered around the Business Analyst role. It's got a short, 1-question poll at the end, so please share your thoughts on it. Thanks! http://mdalums95.wordpress.com/2012/04/05/cio-it-career-project-management-how-to-plan-for-success/

What happened to Requirements Management ?
Does the author assume that 'ALL' requirements have been nailed down.
I've seen some projects where the originator of a requirement changes it
in midstream of the project taking it back to square one (time wasted).
Agile, RUP, CMMI and other process improvement methods address requirements,
but when the man with the money changes requirements, he then has
to reach in his pocket and sometimes he doesn't like to do that !

When you've got 10 spots, some stuff on a topic like this will get left out.

Let's just say I could do a second article just like this, with no overlap...

That said... if requirements change so much that you're "back to square one", there isn't much you can do about it. Either the initial requirement discovery was badly botched, or the business needs have changed so much that the original requirements are irrelevant. That's when "requirements management" can become code for "finishing a project that will be useless when it is finished, regardless of how well it was executed"...

J.Ja

...12 years ago when I needed someone to affirm my observations that multitasking JUST DOESN'T WORK? I'm from the school of thought that doing "one task at a time" allows us to focus on the task at hand and in so doing, we take more time to plan ("measure twice...") get it RIGHT ("...cut once") the first time through...

These are all great mistakes to mention, but changing requirements during the project build phase could be the most pernicious. Unless there is a substantial benefit, project leaders should push to add features at a latter phrase.

Every project requires some experimentation. And I've never known anyone who was able to predict how many experiments would be required to develop a new way to do something, how many potential approaches and sub-approaches may be required in the mean time, and/or experiements to compare various ways by various metrics.

I hadn't heard of the pregnant woman mistake and wish that I had added that to my first book, Why New Systems Fail.

Great list.

Phil

Justin, is this a part of the software engineering myths mentioned in the book 'The Mythical Manmonth' written by Fred Brooks?

We do that now. Management solution to lower the bug count? Take 2 programmers and turn them into CSR's. Yeah, it reduced bug count, because there is less development going on.

Now they are trying to figure out why work has slowed down.

The mistake my company makes (for the 6th time since I have been here), is that it requests time estimates from developers, then routinely overrides them. People are essentially intimidated into giving time estimates that coincide with managements schedule, rather than based on the work to be done. Then we routinely overrun - usually to the "internal" time estimate and maybe a bit more, and morale suffers.

I've recently worked on a project in which everyone was very eager to get a time estimate. I was quite understanding at first because they had just gone through an expensive 2 and half year failed project in which they didn't get what they had paid for in the end. At some point, they had thrown more developers to the project to try to expedite the process.

After the first meetings to get requirements and to understand the scope of the project, I was asked on leaving, "When will it be ready?" I hadn't even had time to make sense of all the documents and notes I had taken. And I was pressed at every turn for a completion date, not for a milestone but for the entire project which isn't trivial.

This created friction. I ended up saying that "If you press for a date, you will get a date. And not much else." I also included an anti-pattern in many official communications. The most common being "Weeks of coding can save days of planning."

I have thus far been able to keep the hostility and intimidation away, but still people are eager to get a final completion date, and as you say these tend to get overridden.

It's really a vicious cycle, one developer disappoints, the next isn't trusted, and ultimately is frustrated and forced to compromise on quality, feature count etc. just to meet a date that pleases management. In which case they may be disappointed by these compromises.

I think the problem is that technology has long been oversold. Computing was thrust upon the business world more by necessity than by design. IT then gave people the mistaken impression that it made things easy, "Just install this, and you get X." I think this has influenced the behavior of executives, perhaps to the point of thinking that accomplishing technical goals is a matter of configuration, not design and engineering. The reason managers want a date is what they understand is they are devoting a set of people to a set of tasks, and it's unproductive to load new tasks on them before they've finished doing the first set of tasks. They want to bring in new clients and new projects, but they have to be able to make promises to make sales, because customers will wonder, "How long will it take, and how much will it cost?" Saying, "It takes as long as it takes," is not going to give them confidence that if they put down their money they'll get what they want.

The dysfunction is driven, I've found, by ignorance, often all around, by degrees. I saw a quote earlier saying (I forget who said it), "The future has arrived. It's just not evenly distributed." That pretty well sums it up, though what that very perceptive person didn't say is "the future" doesn't really get to enjoy it, because "the present," which is a much wider swath of "the distribution" is holding the purse strings, and they call the shots. Some would think that education would help solve the problem, and in principle it would, but the reality of the education system, including at the college level, as it exists is *way* behind on this. That includes the disciplines of CS and SE.

We have built up a culture that believes in using "swiss army knives" (as a crude analogy) to solve problems. We try to take these "standard components" and make them work together to create a functioning whole. It's worked well in some isolated cases for IT, but it very rarely works when it comes to software engineering. Yet we've kept on doing it for decades.

The reality is, though there are people who deny this, we don't have a way of engineering software that is up to modern engineering standards. By this I mean if you compare software engineering to, say, civil engineering, the two are worlds apart in terms of how they think about projects. As someone else said on here, every software project is like an R&D project. That's not how modern engineering works. What I wish the business community and the academic community would admit to is that this is the state of affairs, and some resources need to be devoted to finding some way of doing it that is better. A big part of that is changing expectations on both sides of the spectrum, and that's where I think we'll run into the most resistance. Business people need to understand that computing is a new form of literacy, and it's just as important to understand it as to understand how to read and write. Technical people need to understand that we don't have a science in computing, and we need one badly to support good engineering. Ironically, I think if we started on this, things would appear to get worse before they got better, because we'd start on a process of challenging a lot of long-held assumptions, but the long-run result would be a vast improvement over what we have now.

As a SharePoint Architect I thoroughly appreciate this list. Unlike many of my colleagues who???ve have a strong development background, I???m more business focused; in more of an Engagement Management role to gather requirements and provide the initial translation to a high level solution design before the dev team gets involved.
The point about communications is so important, and while criticism flies around as problems occur, is the responsibility if the business and the dev team to stay in touch, this process is much like a marriage; we all have to be on the same page.
A clear deployment plan is also very important, as soon as the business gets their first taste of the solution the inevitable scope creep starts; ???oh, that???s cool, but can we???.??? But we???re still trying to bug fix.
There so many aspects of SharePoint solution deployment that this list provides a great starting point to keep things in perspective. All too often the kick-off doesn???t set realistic expectations or there are misunderstandings that don???t surface until later in the project, so have a foundational checklist is always a good idea.

Excellent article - these points, all valid, apply to everyone on the team - project manager, developers, business analysts, testers, product managers, etc.

Great list - every item is very relevant, practical, and concise.

Regarding #10 (MutliTasking): what's the difference (not so much in definition, but in impact) of being in a multi-disciplinary role (like yours) vs. multi-tasking?

Refine the project outputs in very clear simple terms,the core resources needed,define the critical success factors,modulisation has worked in the past,as well as dependencies, and time lines.These all were not in place when Bill Gates tried to develope windows Vista.The project was simply saved when Bill was taken off the project.A lesson learned don't be too ambitious.If technical innovation risk is required as part of the project make sure it is tried and tested before building the project.

"Hitting the ground running" also plaques the hiring process. Companies want a new hire to be immediately productive with little, or no, knowledge of culture, procedures, processes, business flow, and/or who-possesses-what-knowledge-skills-and-tools. The next thing you know the new-guy (even a manager) is suggesting changes to processes he/she does not even understand.

Great commentary on the myth-conception of "multi-tasking".

I've also seen "responsibility creep". i.e. a new hire brought in to work on a specific task then diverted, for whatever reason, onto completely unrelated projects. Needless to say it likely spells DOOM for the original project and its schedule.

These ten points are pretty universal so they should be part of any business curriculum. Probably several times.

Sales staff make promises for things we haven't done before, the engineers hand us a stick drawing and the technician figures out some way to make it work in a time frame that has no basis in reality. Only to have the spec change, usually more than once, the engineer draw another line and circle on the stick figure, and once again the tech has to make it work.

I've worked on equipment for earth, water, sky and space and the 10 points are universal. Same song with a different tune. Why should anything with computers be any different?

.

Related to your #10, I worked at a place where I "evolved" into a multi-tasking role. It's not something I asked for. It just became necessary, because we didn't have the right mix of staff to delegate everything that needed to get done. I got a new project manager, after having worked there for 2 years, who assumed I would follow a linear schedule, but that was impossible. He didn't observe or ask me about my work flow at all. He tried to impose one on me, but if I had followed it, some IT operations would've fallen apart. I tried to explain that to him, but he didn't listen to me. He tried to compensate for what I did by reassigning staff to do some of my "diversionary" tasks. My memory is it didn't alleviate my multi-tasking, and he just had to deal with it, or else we wouldn't have gotten anything done.

Just a word to project managers. It's very disconcerting when the software team feels like what you want is out of sync with what they know needs to get done. Depending on the makeup of the organization, people may have found it necessary to multi-task just to keep the whole operation "up." If you're trying to change the work flow by adding/reassigning staff, it's going to take time to do that. "Hit the ground running" doesn't work in this situation at all, and creates disruptions to your entire operation if you insist on it.

Related to your #5 and #6, another problem with large teams is they can miss *huge* tasks, especially if parts of the team are not allowed to communicate with each other, and/or there's dysfunctional communication among the staff (possibly caused by management practices). At this same company, with the same project manager, I discovered a critical task that was not scheduled, and no one was assigned, *two days* before a large system was going to be tested by a client. It resulted from a miscommunication between a software architect and another project manager, who worked under mine. My project manager tried to coordinate a fix for the situation, but did it badly, because he tried to assign blame. I think the reality was it was nobody's fault. The right hand simply didn't know what the left hand was doing. It was a systemic failure. We managed to pull it out, and it was a contractor who I hadn't even met before this who saved our ass. He had been using his experience and a design discipline in his code on the affected system, which enabled me to coordinate a technical solution that a team I assembled could implement quickly, which connected two systems together. It was still a close call, though. We were literally testing and debugging the system, including our "fix it quick!!!" fix, up until 5 minutes before the customer did their first test on it. It was surreal. I always thought those scenes in action movies where something big is gonna blow, the hero pulls it all together in the end with a team, yells "NOW!", someone presses a button, and the system that saves them all starts up, and works, were just fiction! The customer didn't notice any of the chaos, and that's the way it should be, though of course we would've all preferred if the chaos hadn't existed in the first place!

It's amazing how many of these common mistakes boil down to perceptions, or lack thereof, about the development process.

Maybe it's just me, but it's happened to me more than once...
Knowing the products we make and having good requirements for a project, I feel I can make a good estimate hours estimate for a project. However, the project manager seems to think that an estimate of 120 hours will mean the job will be done in three weeks. While I, like many, work more than 40 hours a week, the 120 hours is the time I will be billing the project, not including admin time (manditory department meetings) or support time (since we support our own products).
The project manager for the above 120 hour project was flabbergasted that I told him the development for the project would probably wrap up in four months, not in three weeks. (To note: We were going through a department restructure as well as coming up to a busy support season at the time).