A project manager I trained years ago called me the other day. It was pleasant to catch up with an old friend, and interesting to hear how his career developed over the years. Unfortunately, he did not just call me to catch up. He had a very specific question, and I’m afraid I didn’t have a good answer for him It seems he had just lost his current job because “he did not respond adequately to a production crisis”.

Now, on one hand we could easily dismiss this reason as an obvious scapegoat situation. Someone in the executive ranks felt a need to sacrifice a worker or three so he would not get canned himself. On the other hand we could just say it was this gentleman’s general incompetence which eventually lead to his dismissal; after all, how else would a major production problem occur? Both of these readings, although possible, avoid the question. What is a project manager’s role in a true production crisis?

In some environments, the project manager hold no role in production at all. He works strictly on development and installation, midwifing the changes but never really worrying about what happens next. His knowledge of the stakeholders, the application, and the logic behind why the system functions the way it does never come into play. The operations and support teams do what they can to keep things running and use their own internal systems to respond to crisis situations. In an ideal world, the project team trained the operations and support staff to handle the situation.

Back in the real world, a project manager who remains with a company usually become involved when “his” products take a dive. He is probably already working on another product release, a patch, or just general maintenance. His network of product specific contacts carries information in from the field, revealing potential problems before the support organization knows anything is wrong. The product vendor or developers probably expect to hear from him on a regular basis, whether they want to or not.

Leveraging all of these resources in a crisis can prove dangerous, even fatal, if handled incorrectly. In essence the project manager has the resources to resolve the issue in a timely fashion. However, by doing so he reveals the fundamental weakness of the matrix approach. The project manager wields both influence and power to affect change; the manager retains the authority to make decisions. Rapidly responding to a crisis does not allow the project manager to maintain the illusion of control required to keep this split from erupting into chaos.

This split leads to what I call the “hovering manager” syndrome. The manager attaches himself to some poor senior developer, trying to influence the worker’s decision making process and impress on everyone how grave the situation is. Meanwhile the project manager tries to remove the impediments from his teams path, including sending the manager out to look for left-handed smoke sifters if it seems necessary.

Failing to react, though, can lead to even worse problems. If the project manager simply provides advice, the product could remain down for an extended period of time. The longer it stays down, the more likely it becomes that the executive team will come in search of sacrifices. Once again, the matrix structure will work to the project managers disadvantage – his efforts to get things done will undoubtedly have earned him a number of enemies over the years, leading to his dismissal.

Survival requires walking a very narrow line, deviation from which can lead to disaster. Many project managers I know get sick of it; they leave the profession when flipping burgers or digging ditches sounds better than going to work. Others just hunker down and relish their new roles as project documentation specialists. Most, though, just get right back up and try to walk the line again. I’m not sure if it’s a sign of courage or insanity, but I remain in awe of it either way.