General discussion

Locked

Scary lessons from real-life disaster recovery situations

By Selena Frye Editor ·
This story comes from a Disaster Recovery newsletter subscriber:
"I have been involved with gas explosions, two lightning strikes and the ever common lightning strikes. They do have some things in common: the biggest problem during recovery and post-recovery is ignorant people trying to help. I don't use the term "ignorant" as being stupid or insulting. In fact, I am one of the ignorant ones.
Working as a manager with a local telco company in Boston, MA, I was on the 8th floor right at 8a.m. as the team was getting settled for work, when all of a sudden, a five-story building next to us blew up and was completely leveled. So we evacuated the team, and even though you could see, smell, and hear the results of the gas explosion, some people needed extra coaching to leave.

Well we got everyone out, but given the time of day and subterranean cable vaults below the building, another manager and I thought we should go down and search to make sure everyone was out. Well the gas was very thick but we made it through the vaults and found no one. As we emerged from the basement the fire department arrived and the captain was standing in the lobby, he and his men took one look at us and screamed, "FREEZE." They all rushed us and grabbed our arms and took our 2-way radios away.
Neither of us realized that turning the radios on would have caused a spark and ignited the gas in the air. Despite our extensive safety training, we just did not consider the consequences of our actions. If you need a moral to the story, I guess if you don't know or you are not positive about the outcome of your actions, wait for the experts."

If you have a DR near-miss story, or you can offer a little-known DR tip to share with your peers, post it here. We'll compile a list of your warnings and DR lessons to help others prepare for the unexpected.

This conversation is currently closed to new comments.

6 total posts (Page 1 of 1)  
| Thread display: Collapse - | Expand +

All Comments

Collapse -

Strange

by Tony Hopkinson In reply to Scary lessons from real-l ...

I've worked in heavy industry for many years and one of the first things they drum into you in terms of dealing with the consequences of an accident is don't get accidented yourself.
Other points occur
Were you and your colleague far enough apart so if one was overcome by gas the other could go back and get help.
Did you inform anyone, preferably the evacuation authority of where you were going.
Why didn't you know already that someone was missing or more preferably that you were all safe?

I'd be talking to managment about your procedures in the event of a disaster, in terms of training either you failed to listen to the basics or they failed you badly in not pointing them out.

Your willingness to help is laudable, but your preparation in terms of providing effective help seems to have been woeful.

Heroism does occur in the heat of the moment, so do accidents.

Collapse -

Would you run back into a burning building?

by kevaburg In reply to Strange

I liked the response. I have also been in a couple of situations like the one described and I can definately say that the first reaction should be "get out and stay out".

Electricity and gas make a lethal combination and even turning a lightswitch on can turn a routine problem into a major incident.

It really is time your bosses reviewed their policies on emergency procedures and made sure their employees were aware of them.

Collapse -

No I'd walk very carefully

by Tony Hopkinson In reply to Would you run back into a ...

and be very ready run away like **** if necessary.
Disaster rule number 1
First sound the alarm !

Collapse -

No, don't do that either

by tundraroamer In reply to No I'd walk very carefull ...

I have seen first hand the effects of turning on a light in a below grade room. The outline of the individual is permanently etched into the brick wall including the shadow of his arm going to the light switch. This was a 3 story building, two below, one above and it blew the roof off and killed the unsuspecting worker. This was a pump house in a sewage treatment plant. He never smelled the hydrogen gas. There was no disaster, just a routine check on a pump.
If you are not trained about hazards in modern buildings, then stay out until given the all clear by authorities.

No server room, communication closet or IT job is worth dying for.

Put that at the top of your disaster list.

Collapse -

I take your point

by Tony Hopkinson In reply to No, don't do that either

but I wasn't talking about walking into the problem area to break glass with hammer.
That's the problem when things go wrong, other things tend to follow, unless you take a big step back and some deep breaths any ill/un thought action is likely to have an unfortunate reaction.

Sound the alarm = Run away shouting help by the way. Keeping running until your safe, then help think about what further help you can give.

Collapse -

Emergency Responder Communication

by progan01 In reply to Scary lessons from real-l ...

This didn't happen to me. I was, fortunately, hundreds of miles away when it happened.

A fire broke out on the shop floor of a fireworks manufacturer, in Ohio, I believe, about ten years ago. Per the well-established, well-known and practiced safety routines, the employees immediately closed all windows and shut down the ventilation fans, sealing the vents, and left the building per their disaster plan. Starved of oxygen, the fire would be unable to do more than smoulder, and in time would suffocate itself.

The local volunteer fire department arrived. The senior paid administrators may have been told about the plant's disaster plan for fire. The volunteers apparently had not. One of them broke a window to let the smoke out, standard practice for a conventional fire. The resulting blast killed the firefighters, levelled the building (actually shattered it) and hurt or killed the surviving workers who did not get out of the blast radius. Result: Several dead people, one dead business, and an entry onto somebody's Darwin Award list.

First responders MUST have a plan for receiving and relaying critical disaster information from your business. It doesn't have to be a fireworks factory; if you have servers in the basement a first-floor fire with heavy use of firehoses could flood you out of business. You need to find out NOW who in your local fire department and police department receives and transmits this critical information to the emergency teams who come to save you. Tomorrow you could find out what a fireaxe and a hose could do to your wiring closet.

Call your FD today and talk with them now how to keep a manageable business interruption from becoming a full-fledged disaster. I don't care how much money you put into writing it -- you DO NOT want to find out how well your business resumption plan works in practice.

And if they don't have a means of receiving or acting on your own disaster preparations, you are going to need to adjust your plan to what they're going to do. Don't rely on them doing it your way. If that means putting the basement servers on blocks and putting umbrellas over them, you had best do it.

But find out. Now. The day the thunderstorm takes out power and your business is in the block that's last on their 'restore' list is a day too late to learn they even HAD a 'restore' list.

Back to Software Forum
6 total posts (Page 1 of 1)  

Related Discussions

Related Forums