Leadership

Why does process improvement fail?

It's clear, both anecdotally and objectively, that process improvement efforts have failed far more often than they have succeeded. This blog talks about why this is true.

One of the most famous cartoon quotes of all time is Walt Kelly's "We have met the enemy and he is us." It was first used as a poster for Earth Day 1970, and then later developed by Kelly into a two-panel cartoon. The cartoon's sentiment, while specifically applied to a decidedly environmental message, can be applied to just about any human endeavor.A great deal of effort and resources have been invested in process improvement--whether TQM, CMMI, Six Sigma, ISO 9001, etc--throughout the years. However, it's clear, both anecdotally and objectively, that process improvement efforts have failed far more often than they have succeeded and it's our own faults.

First the statistics

The Juran Institute has estimated that the cumulative cost of mistakes, rework, and scrap will add up to 25-40% of a business' total expenses. The 2004 Standish Group's "Chaos Report," a biannual study based (to date) on surveys of more than 50,000 Information Technology (IT) projects, estimates that only 29% of all software projects succeed, with 53 percent of all projects failing to attain their specified cost, schedule, or performance goals. An additional 18 percent are cancelled prior to completion or delivered and never used. That's an overall failure rate of 71%.

When applied to process improvement, Pareto's principle--more commonly known as the 80-20 Rule--states that 80% of your effort will only produce 20% of your benefit.

There are those who may object to the union of these three approximations of reality. In particular, for some, applying the findings of IT projects performance to process improvement performance may prove particularly galling. There is nothing, however, to suggest that any process improvement project is more complex or difficult than any IT project. Process improvement efforts frequently have a substantive IT component. And more important, both exist within the same environmental and managerial milieu.

To my knowledge there have been no attempts to quantify process improvement success rates. If nothing else, linking the three statistical assertions to the others is as good a starting point as any other, and perhaps better.

The principle of suboptimization asserts that optimizing each subsystem independently, in general, will not lead to an increased optimization for the overall system. The act of subsystem improvement frequently causes the exact opposite of the intended outcome.

In other words, the whole is regularly less than the sum of its parts.

At the heart of suboptimization are some serious operational blind spots:

  • Ignoring the cumulative entropy created by the interaction of the various subsystems with one another.
  • Confusing the maximization of the output of the various subsystems as being synonymous with maximizing the final output of the overall system.
  • Assuming that the final outputs will achieve the targeted goals and/or outcomes.
  • Failing to validate that the targeted goals are actually moving toward the overall organizational vision.

A 1967 NASA report once noted:

"To avoid suboptimization, it is necessary to develop the design criteria logically from the overall system requirements, always keeping the maximum-value goal in mind."

All the engineering mumbo-jumbo aside, the message is: all too often, dependencies between the subsystems are not well understood.

Let's return to the 80-20 Rule. Process improvement has largely been marketed, as well as procured, as an enterprise-wide solution. Mostly, this approach has been a reaction to the serious flaws of IT management in the past, where the organization inadvertently adopted a flawed or incomplete re-engineering strategy, and too little attention was given to the elicitation and validation of requirements. However, the enterprise-wide approach has some serious flaws: it assumes that everything that is planned can be realistically achieved.

Not all processes are appropriate targets for process improvement, because the cost of the improvement may be more expensive than the increased productivity they generate. Moreover, the 80 -20 Rule is, in effect, an inefficiency constant. It tells us that an across-the-board enterprise process improvement approach will in all likelihood result in failure. If 80% of your effort results in only 20% benefit, then that only leaves the remaining 20% of your effort for the remaining 80% you seek. You'd find better odds in Las Vegas.

Using the 80-20 Rule as a starting point, we can assert that 20% of a company's processes produce 80% of its waste and rework. Jay Arthur of LifeStar has suggested a different, more focused approach. He points out that the 80-20 Rule can be applied to itself (i.e., 20 percent of 20% and 80 percent of 80%), resulting in 4 percent of a company's processes produce 64% of its waste and rework. And applying this approach one level further results in a rounded outcome of: 1% of a company's processes produce 50% of its waste and rework.

In fact, process improvement efforts often trigger increased internal competition for scarcer operating resources, which in turn may act as a catalyst for unexpected personal, business unit, and cultural conflict. It may also produce unexpected bottle-necks in organizational decision-making. These unexpected consequences of process improvement implementation must be dealt with by an organization that is already stretched thin trying to change while still maintaining the functioning of day-to-day responsibilities.

Yet business executives and senior managers continue to pursue the "heroic" approach to process improvement. Why? There is, I'm afraid, no good answer.

15 comments
jmgarvin
jmgarvin

nAch is a huge factor in why process improvement fails. The problem is that humans aren't factored in. nAch (the need to achieve) is ignored. Another fail point is need. Why do we NEED strong processes? If it doesn't come down from on high, it'll never succeed.

Richard.Miranda
Richard.Miranda

I used to work for a company that adopted the Kaizen approach to process improvement with great success. Rather than massive re-engineering efforts it employed an ongoing continuous improvement approach. A whole lot of small improvements over time worked better than trying to alter things on a large scale. Since this was integrated into the company's overall philosophy it required buy in from the CEO all the way down the food chain. Continuous improvement teams were formed from people outside of a target department to maximize a fresh perspective. They were, however, to work with the department throughout the process so as not to eliminate any essential sub-process and to minimize the culture shock from the changes they instituted. The team was aloud only two weeks to analyze and make they're improvements so as not to get over zealous. The overall Kaizen effort was coordinated by a continuous improvement department (about three to five employees) and the team was dispanded and returned to their normal duties after that time. For the most part people enjoyed working on these teams since it gave them a break from their usual routine. The target department got a lot of their gripes listened to during the analysis phase and a good portion of them addressed. Since this was in the aerospace industry where fear of layoff is always present, one of the conditions set down by management was that no employees would ever be layed off as a result of any improvement in efficiency made by a Kaizen effort. These projects were to increase efficiency and productivity. If, for some reason, a position was eliminated, and this was discouraged, the employee was guaranteed another position of greater or equal value within the company. If a position could not be found then one would be created within the continuous improvement department. In the two or three remaining years with the company I never heard of anyone losing a position because of this. Morale was improved and productivity was increased on a departmental and eventually enterprise-wide level as a result of this approach. My experience with this has led me to believe that this approach of continuous small improvements is far better than any massive re-engineering effort.

martygrubin
martygrubin

I have lead and been part of several process improvement projects and here are my observations on why they sometimes fail: 1) No organizational change management; the workers must be part of the solution 2) Devotion to one methodology that may or may not be appropriate to the case (e.g., Six Sigma in knowledge based processes) 3) Operating management not fully engaged in the outcome, just paying lip service. 4) Avoiding the details of how work is actually performed; too many BPA analysts do not want to get their hands dirty. 5) Plain old inexperience; there are a lot pretenders without any real projects under their belts. I have seen these projects produce spectacular results for companies willing to do devote the resources.

mikifinaz1
mikifinaz1

Most people fear process improvement and when directly asked will just give you platitudes so that they can't be singled out as not being a team player. If you don't have the people onboard they will kill it. Because they: see it adding useless work, management uses it as a stick, the workers feel it is going to lead to their downsizing etc. etc. and management is usually hostile because they don't see the ulitimate benefit and think it is going to bankrupt them. It takes a true statesman to pull this off in most environments. The best bad example I saw of this was in a games company and the best example was Microsoft. In the good example the dept. manager used a couple of simple projects to show everyone how it works, why it works and the benefit. Then he had the support for doing it live on a major project. Once everyone is onboard enough to give it a try, any problem with the details will be overcome creatively and quickly.

jck
jck

In my experience: Quite often, management pursues technical utopia. That is, they expect humans to be robots. They expect machines not to break. They expect kids not to get sick. They expect...well...too much. Big mistake. Improving a process by expecting your employees to be more efficient when they are already being quite efficient is futile. Rather than to have meetings and waste that time figuring out every little tiny inefficiency, you're better off scheduling in some slack time and letting your people be people. Also it seems that in the quest to be ever-efficient, a lot of organizations cut corners on process which leads to things like product quality drops and workplace mishaps occur. Management and Executive level planning needs to allow for humans to be human. It's one thing if you have a programmer taking ten 10-minute smoke breaks in an 8-5 day. That is wasted time and can be adjusted for. But it's another to stand over your people and point out every little flaw or inefficiency in their work process...whether in person, or in policy. That not only brings micromanagement into the picture, but it often starts to pick away at the morale of your workforce when you do. Process improvement is good, when done in a balanced way. But often, it is taken to the extreme to bleed every extra second possible and remove any step not absolutely necessary. I think that's what causes failure. Basically: the attempt to bring absolute order to a innately chaotic world will never work.

forrest
forrest

I agree with all the noted points; however, I believe that there is an answer. Business executives and senior managers have felt that they needed to do something; hence, they did what they thought was right to improve their metrics ??? even if that led to suboptimizations. Until recently business executives and senior managers did not have a system for orchestrating the enterprise as a whole and moving the organization toward achieving the three Rs of business; i.e., everybody doing the Right things and doing them Right at the right time. But things have changed. Today the Integrated Enterprise Excellence (IEE) system, an enhanced application of Lean Six Sigma at the enterprise level provides the framework and roadmap so that organizations avoid process-improvement suboptimizations. IEE a structured approach for moving toward achieving the three Rs of business through the wise blending of measurements, analytics, innovation, process improvement, and control at all levels of an organization. Forrest Breyfogle

PMPsicle
PMPsicle

Yes there is a good answer .... we've become a society of "the quick fix" aka "the magic bullet". Meaning people don't take the time to critically assess solutions. There is nothing wrong with process improvement, per se. There is, however, two serious issues with the "heroic" approach as the author refers to it. First, risk (or more properly downside exposure). The bigger the scope the higher the risk (meaning probability) of failure AND the bigger the effect (by definition in this case downward). Thus not only are the failures more frequent they are more spectacular. And since one company only goes one way or the other by definition we hear more often about the spectacular failures. Secondly, change is usually poorly handled. And the bigger the area of effect the more poorly it is handled. Unfortunately, the bigger the impact the more important it becomes to handle the effect. Even if risk were not a factor the increased difficulty and decreased effectiveness of (organizational) change management (aka reaction management) would doom most "heroic" business improvement projects. Glen Ford

robert.leap
robert.leap

A husband's wife fooled around his back and when caught ? she blatantly told her husband, "your process definitely needed improvement ? so I outsourced!" Needless to say, the husband is a bitter man.

larue
larue

the openness of managment. I worked for a company that tried to cross over into the lean world of mfg. And let me say here that, Lean is an awsome awsome concept/way of life. They spent/wasted millions trying to make the transition. When a company has a, your words, " all the way down the food chain" mindset, it is most likely that failure will prevail again and again. I like to phrase it "across the board", or equals. How can a body function without all the parts, is one part any greater than the other? I think that until a company begins to look at the world this way, they will continue to struggle. Buying in, is the MOST difficult thing for managment. Way more than for (the bottom feeders) as i have heard parts of the interprise called. Its ok to joke around with things such as these, but when the poop is headed for the fan, there had better be a geninue RESPECT, understanding and belief that all are equal. I say its harder for managment than those on the shop floor, because normally much more is invested. IE Education. Unfortunately, for many, along with Education comes a holier than thou attitude. Lean is a way of life. its about replacing the complex with the simple. Kaizen is a part of lean like the hands are part of the body. The shop floor must be an absolute EQUAL part to the venture. Think of it this way. The heart is the company itself, and yes it has a more important roll. for without it the company is dead. Managment is the brain, but not fully because without the eyes and or the rest of the body,(the shop floor) providing information to the brain, the endeaver will not see reality. You really cannot have one without the other, it makes up the body, or the business, the Interprise. All in all, many things keep American companies struggling to compete, but the most outstanding thing i have witnessed over the last 38 yrs in mfg, is the seperation within each company. I will stand down from the soapbox at this time for another. thank you. Roger Smith Owner Design Welding Inc.

donstrayer
donstrayer

Kaizen works because the improvements come from bottom up rather than imposed. Most professionals really do want to do their jobs better and when empowered to do so they will. Management must foster this effort, but not mandate solutions. This is not to say that Kaizen teams should reinvent the wheel. They should incorporate best practice models such as CMMI and ITIL where applicable. They should be trained in both the applicable models and in process improvement techniques. All too often process improvement efforts fail as soon as the sponsoring manager moves on. Months later we come to realize that we really needed it and we start over. And we fail again. How many times have you baselined your IT assets? How many times have you implemented a project portfolio management system? SDLCs? Scorecards? Metrics? Project reviews? QA? If you've worked for the same organization for several years you've probably seen many of these and others done, abandonned, then redone more than once. Top down process improvement efforts begin with a vision of the future state followed by a mandate to make it happen. Kaizen begins with a trust that empowered and trained subject matter experts can devise and implement continual, incremental improvements we may not have envisioned. We don't know what the future state will be except for implementing and fostering Kaizen. This is a difficult change for many managers. We are, after all, the leaders and the experts. That's why we're paid the big bucks, isn't it?

scmarkley
scmarkley

Im no expert in this subject but I have over 30 years experience in the technology business and have some ITIL certifications. I agree wholeheartedly with jck and feel that the root of the problem is unrealistic expectations such as the "silver bullet" and the the utopian business model. The failure to get real is the real failure.

M.W.H.
M.W.H.

Sorry... nothing happened when I visited it but I have removed the link just in case.

sdsondhi
sdsondhi

I agree with the fact that change is handled poorly. I think the most relevant point is that the agility required to nurutre change is often balanced against profitability because a company calculates ROI prematurely rather than in retrospect as it should be.

ghobday
ghobday

1. While there are no silver bullets, there a lot of people selling them and finding buyers. Remember snake oil. 2. We have a Harvard MBA ethos which promotes and translates 'control' from a high volume manufacturing model into every kind of business. 3. The world really is a lot more complex and changing than it was even 10 years ago. I think we are collectively and individually reaching the limits of our capacity to process enough information for good decisions. And everyone else in our decsion group has lots of (different) information. 4. Lots of IT solutions and process solutions really do work and produce good results. Can it improve the overall business in isolation or in another context? Only with a lot of fine tuning. Lots of good, sound ideas, fail in execution because no one (in our culture at least) recognises the effort needed to fine tune and carefully knit the new piece into the rest of the fabric. 5. I think applying 80/20 to 80/20 is an example of taking a simple idea too far and just adds to the complexity. 6. What we are all trying to do is rebuild a ship at sea. The weather is getting rougher and the ships bigger. Smaller boats may be the better solution to managing business risk than an IEE blueprint for the super tanker.

beaudop
beaudop

Thanks for this link - when I went here it tried to install something on my PC.

Editor's Picks