Focus on the Main Body

In my current work environment we have very stringent requirements on ceritifcation of new technologies. These requirements are driven by the customer and non-negotiable. Its like humidity. Its just there.

Frequently technologists here, given a problem, look to vendors for ideas on how to solve the problem. All well and good, except that the vendors INVARIABLY drive our technologists to newer versions, which are frequently not certified for our network. The certification lag is routinely 18 months. Again we have no ability to effect that.

The problem we enoucnter is that developers generally want to use the newest technologies, generally for excellent technical reasons. The fact that these technologies are not available in our environment generally doesn't faze them.

This year we encountered a cost reduction project which was designed to shave $1 million a month off of our costs. The development team chose to use Windows 2003 for IIS 6.0, even though nothing in the technical solution required it and the proof of concept was done on W2K / IIS 5.0. After four months of development the customer has rejected the request to release to production because W2003 has not yet been certified. That process has 6 months or more to run. The net cost of the decision to IIS to my employer will probably be $10 million in 2005.

The technical recommendation was correct and accurate, but the business decision was wrong.

With dwindling IT budgets, stiff and increasingly skilled off shore competition, what is an IT professional to do to maintain the illusion of job security?

The answers seems to be similiar to investment advice. If you don't know an insider who will whisper the 21st Century equivalent of "Plastics" then you should consider diversification.

The team that I work on has about 8 people. 5 of these are "Oracle DBAs" or "Oracle Engineers". For the most part they are technically competent and able to contribute to solving Oracle problems. The other three are uber-Geek wonk-heads who treat each new request to work in a new product/environment/area a wonderful opportunity to learn. Intrestingly enough, both groups will consult the same books, websites and vendor resources to solve Oracle problems.

The uber-Geeks will apply the enthusiasm and desire to SQL Server, MySQL, Sybase, or flat file conversions for that matter. In the inevitable contractions in the workforce it appears clear that those folks with

1. A diverse skillset.

2. A yearning for new knowledge.

3 A passion for IT technologies in general

will thrive. The one-trick ponies (no matter the pony) will be eventually left standing when some music stops. And no, product upgrades DO NOT represent diversity.

So, just like your 401k, diversify your skill set. LEARN that new product. In all likelihood that will be ticket to keeping a good job or finding a better one.

Scenario: You have just been given the opportunity to lead a team of technical professionals who have designed, built and operated a critical support system which, by all accounts, is under performing everyone's expectations. The team is located in three geographical areas. One does engineering (think new products, patches, upgrades) and the other two do operations.

90% of the team has been on the project for 4 years. Last year the engineers spent 1.5 man years "engineering" a product (version) upgrade for 12 instances. All of the instances failed to upgrade during the scheduled maintenance windows. The failures were related to things like missing installers, low disk space and missing patches.

Your mission: Transform the team from "the gang that couldn't shoot straight" to "database delta force"

The constraints: You have 8 people on the team. You can only grow to 10. It takes 6 weeks to get a new hire into the environment and work (the job requires a security clearance). The corporate culture will only allow raises of about 5% a year.

What we did was get everyone together. Let them vent for a day and then explained the new vision. We didn't dwell on the mistakes of the past, but rather looked forward to the future. We explained the standards that we need to be met and gave everyone ample opportunities to meet them. In fact, the standards weren't all that high. The engineers had to engineer for the production environment and the production operators had to document all changes in the enterprise Change Management system (Remedy).

At six months into the exercise here's where we are

Team strength - 7: We lost two senior operators who didn't want to adapt. We let go one sub contractor who couldn't meet dealines. We gained an internal transfer and 1 outside new hire.

Performance - Overall the teams' performance is about like it was before, but at almost 20% less cost. And with the new hires we are poised to make some improvements.

The basic methodology has been to document (primarily in focused emails) the work standards, and the follow up, follow up, follow up.

I would rate our success at about a B for the first 6 months.

Been there, done that?

Its 2 AM. The business critical database has been down for 8 hours. Your team is fading fast. The vendor is in a "post your problem to our website, we'll post a solution back" mode? What are your options?

First - if you're asking that question at 2 AM you're already hosed.

Critical incident management, or whatever phrase you use, is not something that you can invent on the fly. There are well known patterns and practices for incident resolution. ITIL is an up and coming collection of best practices and provides a library of ideas. Its a good starting point for know what you SHOULD do.

The HOW you do those things are up to you.

Like most important things, the time to start critical incident management is well before the outage. You should have a core of people who are trained in how your organization manages critical incidents. This team should be able to support 24x7 management of critical incidents for up to 3 - 4 days. The number of teams you need will be determined by how large your enterprise is.

The primary talents team members need is the ability to create and maintain an accurate picture of the situation and the ability to keep a diverse team focused on specific actions. Your incident managers (IM) do not need to be deep technical experts, but they MUST be familiar with the technology and the environment.

Where can you find IMs? The IM factory is the United States military. Find a senior NCO or mid-level officer who has stood watch in a TOC during a battle (real or simulated). These individuals are adept at managing multiple things and disciplined enough to keep good notes.

With respect to the notes, this is a key element. A major incident will involve one or more vendors. The vendors will have varying degrees of commitent to resolving the incident. It seems like their first priority is to blame any technology that is not their own. The notes will become critical as the vendors and your IM team go through shift change.

The notes aren't novellas, or event blogs. They are log entries. Date/Time (including time zone), and a brief note. Most of the notes should be in form of action items. Resolving the incident quickly is all about action.

An action item is something, generally a task, that has a number of properties.

1. Identifier - Every action must be uniquely identified. A numbering system like A.1, A.1.1, A.2 is common. Regardless of the system, each action MUST have a single way to reference it. Your IMs HAVe to be able to communicate in terms of "What is the status of Action Item A.1.1?" and have confidence the answer back is really about that action item.

2. Every action must have a Next Action. This should be short and defined. (restart server, restore /2a/datafile31.dbf from tape, etc) This allows for chaining of a sequence of steps to complete the action item.

3. Every action item has EXACTLY one owner. The owner is a PERSON. The owner MUST accept the action in a conversation or email with the IM. Action items CANNOT be owned by Organizations, groups or pairs (ie, Bob/Bill. Who is responsible???)

4. Every Next Action must have a date / time for status. This date / time MUST be provided by the OWNER. This is a primary component of accountability.

Following the rules for action items above will decrease the time spent on negotiating this information after deadlines are passed and opportunities missed.

IMs should collect all the action items in a single document, called an Action Plan. The Action Plan should also include a section for references (phone numbers, ticket/case numbers, etc). IMs should include relevant discussions including the date/time and noting action items which resulted from the discussion.

The Action Plan should be electronic. This will allow IMs to pass it during a shift change. You can create a web application, but a text document works ok as well.

During the incident communications are VITAL. Companies that can should stand up a conference line for the duration of the incident. This will ensure that everyone can communicate quickly and the IM can keep her finger on the pulse of the activities. This reduces the number of side bar conversations that can lead the technical team astray.

Finally, the IMs should be insistent on the technical team members completing DEFINED actions as specific TIMES. This will allow the team to continually move forward.

I'll have more suggestions for incident management as soon as I get some rest.

You see, we had a 60 hour outage in a business critical system this week. About of third of that was wasted time with the incident team spinning in circles. I was really the only IM we could find between three organizations with a total of about 60 folks.

So its been a long week.

So. The transformation experiment failed. I was unable to convince my manager/friend to enforce the standards required to lift the organization. We devolved, rather quickly, into the kind of organization, where a few people worked at 110% but most worked below 33%. As is inevitable, we burned out the 110% components.

The root cause is correctly put on the leadership. However that oversimplifies the issue. In fact, it was the leadership's failure to clearly communicate the gravity of the personnel resource issue early enough to prevent burn out. Even though I work in a global company with enarly 100,000 employees world wide, for the first 3 months of our efforts the only resource available to me was my Rolodex. This didn't work.

Finally, last week, after months of no change, only more work, I left the manager I had been with for 2.5 years. Now I am an individual performer, working on tools. Which is fine for me for a while.

Here are 10 things that you should know before you starting coding....

1. Know the customer. Know why they want the product. Know what business problem it will solve. Know what external forces may cause the problem to change (regulation, executive direction, etc). Know how they will measure its value. Know how they will know if you delivered the value you committed to.

2. Know the user. The user may or may not be the customer. Users and customers have different perspective. Know how the user does the tasks today. Know what tools and technologies she employs and which one work now and which ones don't. Know how you can make her job easier and still value her contributions.

3. Know the production environment. Know what technologies are available. Know and be able to measure, the cost to implement new technologies. Know which systems are reliable and which are not. Know who can effect change in the production environment and how you can communicate with them. Know the roadmap for the production environment and align accordingly.

4. Know the development environment. Know how to access your version control system. Know how to access the defect tracking system. Ensure you have accounts in all of the systems where you will work.

5. Know the development team. If you are an individual performer, meet each team member individually. Schedule an informal phone call or drive by to their workplace. Know what technologies they are most comfortable with. Know what they want to get from this project. Be sure to include the testers / QA / project manager(s). If you are a lead, know the talents each team member has. Know what kind of learners they are (visual, auditory, tactile, etc).

6. Know the design of your solution. Know the architecture and which design patterns are involved. Know each component of the design and how they interact. Know which messages get sent and what returns are expected.

7. Know the inputs and outputs. Know how the system will recieve input and generate output. Know the interfaces for the external systems. Know the frequency for inputs and outputs. Know the permissions you will need on the output system and test them.

8. Know how to measure your solution's baseline. Know how you will measure your solution's performance. Know how to make discreet measurements for the components you build and how to isolate your system's performance from the input and output systems. Know how to measure which users use which features and how long it takes them to use those. Know how to turn this off and on as needed. Know how to analyze the data to find problems and isolate the source.

9. Know when you're done. Know what it takes to be done with the project. Know what the requirements are and how you will demonstrate each to the customer. Know how you will recieve defect feedback and know how you will deal with different kinds of defects. Know what performance measurements mean your system is working.

10. Know how to communicate. Know what the standards are communicating with the customer, the user, and the team. Know how to write grammatically correct business language. Know when, and to whom to escalate. Know how to formulate communications so that the recipient can answer "Yes" or "No". Know how to track communications for future reference. Know when a phone call or walk by is more appropriate than an email or other written communication. Know that anything you type, and some of what you say can be used against you in a court of law, design / code review, late night bug fix session, performance review, exit interview.

NOTE: For purposes of this conversation, KNOW = document. Additionally, there are references to things in this posting that many developers and teams don't do. (Such as know what design patterns are involved, etc. If you don't do that now, you should. The same goes for defect tracking)

A colleague of mine recently forwarded a diagram that had been produced by person on the account where he now works. The purpose of the diagram was to allow the account management to show the customer how the account team planned to resolve a significant impasse in getting data from one subcontractor's system into another subcontractor's system.

The diagram had an number of nice colors, and arrows connecting the approrpriately labelled boxes in each subcontractor's domain. The only thing missing were the ports and protocols. You might think this was an oversight, however, as it turns in every case at least one subcontractor blocks a port / protocol that could be used to transfer the required data. In most cases, both sides blocked it.

At this point it became obvious to me.. what was needed a is port / protocol combination that technologists, faced with a series of non-technical roadblocks, can use to accomplish the impossible.

Hence, I am proposing the new Multi-Application Graphical Connection (MAGC) protocol and by convention we'll use port 10 (from the old joke about how many kinds of people understand binary).

HTTP blocked? No worries, use MAGC on port 10.

SMTP blocked? Never fear, your mail message will make through using MAGC.

FTP disabled? Not a problem. We'll just use MAGC to automate that file transfer.

And appropriately enough this is by far the most common protocol in business logic diagrams. Particularly those used to commit tens and even hundreds of millions of precious IT dollars.

So.. the next time you're faced with a hard deadline and the management is screaming for that connectivity.. reach into your developer bag o' tricks and pull out the Multi-Application Graphical Connection (MAGC) protocol.