CXO

Cleaning our own house: Serious need for process improvement in IT

It's hard to undertake an organization-wide business process improvement effort once it's discovered that your own processes are something of a mess. One CIO discovered the hard way just how deep the problem was in his own IT department and is taking action to correct the problem for the good of everyone. Benny Sisko tells his story.

It's hard to undertake an organization-wide business process improvement effort once it's discovered that your own processes are something of a mess.  One CIO discovered the hard way just how deep the problem was in his own IT department and is taking action to correct the problem for the good of everyone.  Benny Sisko tells his story.


Where I work, we're undertaking a business process improvement effort for which I'm leading the charge for the organization.  Our effort actually began last summer, but the "perfect storm" (described below)  took place, successfully derailing our efforts.  Before things got derailed, we were making some real progress in the identification stages of the project and had a good plan for the future.  Of course, as they say, the best laid plans of mice and men often go awry.  Leading a small department - we had eight people in IT at the organization at the time and now seven - is difficult in the best of times; small IT departments still have similar demands as large ones, but with fewer people to handle the requests.  My team, however, is very, very good and I felt that we could make small steps forward with business processes while continuing our daily operations.

The Perfect Storm

Then all hell broke loose.

In late July, my second-in-command and fantastic database professional left to take a higher paying position in another organization.  On top of that, his backup went out on maternity leave shortly thereafter, leaving a major gaping hole in the department.  That, plus a major unexpected support burden over the summer led to what became, by far, the worst few months of my fifteen year IT career.  Forget process improvement; IT was barely able to keep up with support requests, which absolutely exploded.  From the months of September through December, we barely kept our heads above water and, on occasion, swallowed huge gulps which left us gasping for air.  The unanticipated departure of a key player followed by the temporary loss of his backup (we do cross-train) left us in bad shape.

(Before I continue on with the process improvement part of this, I want to tell you that the person that left for greener pastures ended up coming back to work for me in November after deciding that his new job did not leave him satisfied.  This is a person that thrives on challenge and change; in his new position, he was simply bored.  Of course, after a couple of months, his backup also came back after having her baby, so the team from last summer is again in place and we've managed to work through the mess and get on solid footing again.)

Life Goes On

Now, on with the story.

I described these few months as absolutely horrible; this is true.  But, I also learned a whole lot during this time.  In short, while I was looking around the organization for processes to improve, I did not know the depth to which our own processes needed serious attention.  This lack of knowledge was a good portion of the reason behind our fall struggles.  Although we had some documentation related to dozens and dozens of processes in which IT gets involved, said documentation was incomplete at best.  I don't lay the blame for this lack of documentation at the feet of the person that left and then came back.  It's really a continuation of what took place when our ERP was originally installed ten years ago.  At that time, not a lot was documented related to pretty much anything and documentation has been built in just the past few years - I've been with the organization a bit less than three years.  Now, I will tell you that I knew our documentation was far from perfect and that our processes, while getting better, were still semi-manual.  However, when that gaping hole in our database skill set opened up this fall, I was unprepared for just how bad it really was.  For example, where some processes should be solely under the control of end users, IT has to run a manual script to kick it off.  Where a complete process should be totally automated, we still have in place yet another script without an automated job tied to it.  Where documentation should be complete, we are still missing key pieces of information.

How did it get this way and how could I not know the severity?  There are a couple of reasons:

  • Focus. I stepped into a real mess when I assumed the CIO position at my current employer. Infrastructure was in really bad shape; IT morale was in the toilet and the rest of the organization saw IT as a roadblock rather than as an enabler. My first 18 months were focused mostly on fixing infrastructure issues on which everything else would ride. I also spent a lot of time playing politics, getting IT to be seen in a more positive light. My data person simply handled the data tasks that came his way. While I knew was he was working on, it wasn't my focus and I wasn't getting complaints from his customers. As such, my attention was focused elsewhere. Yes, I knew we had weak processes in that arena, but they were working, unlike a lot of other things.
  • History. I let us be controlled by the "we've always done it that way" syndrome - at least on the data side. On infrastructure, we pretty much threw everything away and started from scratch, but not so for the data side of the house, which is much harder to change quickly. We were still living with processes developed ten years ago that had never been reviewed or changed. The CIO originally in charge of installing our ERP made heavy, heavy use of custom fields, but kept the documentation for how these fields work and what they do in his head - none of it was on paper. When he left the organization, the DBA he left behind (who I mentioned earlier) documented as much of it as he could and, as he was able, continued to discover things along the way.

A Refocusing

So, what are we doing now?

  • Documentation. I don't care how long it takes or if developing proper documentation results in missing a deadline by a few days; as new items are undertaken and as we revisit existing processes, they are being documented from start to finish. No more incomplete documentation that gets us halfway there! My goal: The next CIO that sits in my chair can walk in day one and know exactly how things work.
  • Revamping. With infrastructure now solid and secure, I'm focusing much more on the data side of the house, particularly now that I know just how exposed we are on that front. For example, I discovered last year that a significant chunk of our problems were due to poor identify management efforts; a main focus for us now is to implement a real, complete identity management solution that encompasses all of our systems. My goal: Don't make IT get involved in every single process just for control's sake! My predecessor was guilty of this sin. Again, IT should not block routine things from happening!
  • Centralization. We have some processes being run through SQL Server Integration Services, some running through Windows scheduled jobs and some run completely manually and requiring twelve steps to complete. Over time, we'll consolidate 100% of our own processes into a single standard operating procedure, based either on SSIS or BizTalk. My goal: By centralizing our processes, much less discovery is required to find out how things work. This, coupled with good documentation, will make life easier for everyone.

The last few months have certainly been a period of education for me.  While extremely stressful all around, I learned a number of valuable lessons.  Without my "crutch" in the data department, I was able to see where our true weaknesses lie so that I can better develop a plan of action for correction.

About

Since 1994, Scott Lowe has been providing technology solutions to a variety of organizations. After spending 10 years in multiple CIO roles, Scott is now an independent consultant, blogger, author, owner of The 1610 Group, and a Senior IT Executive w...

2 comments
wesknox
wesknox

I spent a number of years in an IT organization where you couldn't spend 10 cents for off-the-shelf software but you could spend 2 people's salary for 5 years to develop it internally. I learned the documentation lesson very well over that tenure. If it's not documented, it's lifespan is limited to shortly after the author/developer leaves. This is a recurrent ax that I seem to grind all too often...the poor job typically done by organizations in capturing new situational knowledge and extending (or even constructing) their knowledge base. If you have spent any time on a technical support line, suffering the slings and arrows of outrageous support frustration, you likely have encountered this. You get past the 80% 'dummy problem' list with 1st level support...maybe even 2nd level support...to the "engineer(s)". They fairly quickly understand not only what you are telling them, but also what the causes are/may be and either tell you it's NAPWAD (not a problem...works as designed), or correct it for you. NOW...and where the rubber typically doesn't meet the road...the knowledge base is not amended to include 1) the key distinctive piece of information (trigger) NOR the corrective process to resolve the issue. Now, think about the cost/value of this knowledge...Frustrating (certainly) hours (probably) invested by the customer to wade through the diagnosis tree (times 2 accounting for support staff), interruption of engineer's time to participate in 3rd level support...customers who bailed out waiting on hold (many of whom are victims of the 80% problem for whom the whole support solution is fine-tuned for!) It's stunning to think of the amount of wasted human effort from the fundamental inability of organizations to "learn" from their own experience (both mistakes as well as successes). Many overlook the fact that learning is not just committing something to human memory. For an organization (and actually for an individual) you must commit information to longer term, more reliable memory...especially if it's important, critical, expensive-to-come-by knowledge. For an organization, this knowledge needs to be organized and efficiently retrievable (or else it's not worth having). Ask yourself why Google has been so successful at it's core business. It purports to provide you access to information quickly and efficiently across the "known universe" of information published to the web. It's been my experience that two methods occupy the endpoints of the continuum of solutions. On one end is a highly structured, organized, knowledge database that enforces an organizational heirarchy to the information (and possibly a component designed to retrieve information based on a step-wise Q&A traverse of this knowledge tree). On the other end is the Google/X1/Copernic solution of brute force indexing of everything and providing content-search tools to retrieve information. In any case, resolve to improve the capture and organization of your personal and business knowledge assets...goodness knows enough money has been spent to acquire it. Otherwise you just may spend an hour or more looking for your car in the stadium parking lot.

laura
laura

Thanks for sharing your story. I've worked in similar situations. It's amazing how basic processes can introduce tremendous operational efficiencies. I am sure your staff is happier as well, given the opportunity to "do things right." Best, Laura Brandau

Editor's Picks