29 replies
by dc_guy
To fully prepare for a disaster of this magnitude, you must have a full disaster-recovery plan. You can't possibly be ready for a volcano eruption that close to your data center without it. Depending on what business you're in and how much down time it can withstand, you may need a "cold" site or even a "hot" site. You'll certainly need off-site backups.
Sitting there in your comfortable chair on a nice day, you can probably do a halfway decent job of imagining what will need to be done if the volcano erupts. But when the eruption happens and everyone is thinking about their homes and families instead of their jobs, no one -- including yourself -- will be able to think clearly.
You have to have a detailed plan to follow, and you may even need to hold practice drills.
I've been through a few earthquakes, so I know what I'm talking about.
I have to do DR for the Gov, including disasters, acts of God etc etc. One thing management and auditor's seem to forget is that Computers run on electricity - if you don't have a generator then no matter how good your DR is - if you have no electricity you are stuffed.
Also they forget about the Customer's, Suppliers etc - if they have no DR - their will be no deliveries etc etc
Sometimes you need to think laterally and practically.
One thing to also keep in mind is that generators need PLENTY fuel to run. Most large generators get terrible fuel mileage in that they might run only 4 hours on a full tank of fuel. This means shutting everything down to refuel the generator.
You can get generators that run on diesel and keep a bunch of 55 gallon drums of fuel on hand. Also remember to get a hand pump for transferring the fuel into the generator.
When Hurricane Isabel hit last fall, many generators ran out of fuel after a day or so and the power was off for over a week in some places. Gas stations need power to pump and many of these were not open for several days up to a week.
by tundraroamer
This is not my first volcano experience if it happens so I kinda know what to expect from this particular volcano. And earthquakes are a way of life here. However, our network and branch locations have grown larger and we are more dependent on our computers then the last time 12 years ago.
I have thought about power issues, people, transportation issues, backups, air filters for both people and equipment but its what I haven't thought of that bothers me. So that's what I am looking for. People that have gone through a natural disaster and later found out what they should have done prior to the event. I think that's the part of the disaster plan I am missing.
by dazau
I recently visited a volcano in Vanuatu and was amazed at the amount of ash that was raining down from a volcano 6 kms away.
You may need fresh water if the catchment is polluted by large amounts of falling ash. Are you using any form of water cooling? Does your generator have a water cooled radiator that may become blocked with ash? Broom/brush etc to keep them clean.
You may also need showering facilities for your key people if they don't want to be all dirty. Some people (esp the opposite sex) can't handle grit and dirt in every orifice and you may need to rely on these people. Keep em happy if you want them to work while everyone else goes home to help their families.
If you have volcanoes then you may have earthquakes/siesmic activity. Do you have backup comms links?
If the ash gets really heavy (was like light rain where I was) then any form of radio or wireless comms may be affected too.
If the ash gets really heavy then it may get dark - need torches/floodlights etc.
And the obvious question - do you have accurate warning system and information when/if the volocano blows. The web is full of monitoring stations and most active volcanoes have monitoring equipment too. Search Google for "volcano monitoring" if you dont already know a good site.
PS: If your trees end up covered in ash then just give them a swift kick around the base! (Works in Vanuatu!)
by rmrwork
Geez... a volcano. Yikes.
First off, if you don't have a pre-defined communication plan in place, you should do that ASAP. Be sure to define a procedure for both internal communications and external communications (depending on what sort of company you work for, you may specific obligations that need to be met by you or someone in IT for external communication to the media, government, etc).
Relative to Internal communication, you need to define who in your organization has the specific authority to implement the execution of the D-R plan and begin the recovery / contingency plan that was built. In the event of a disaster, it is key that employees and contractors affiliated with your company know who will be giving the "go" command.
If you do have an existing plan, take a "walk-through" to verify all contact information is current / accurate.
Once the decision to "go" has been given, there should be a clear path of communication to ensure that the plan is executed as planned.
This includes communication with your end-user / client community that relies on your systems. They, especially, need to know who to talk to and who they should be taking their instructions from.
Also, you should include a cascading notification process to notify / activate your key IT or end-user personnel. As noted by one of the other correspondents, they may very well be pre-occupied with the well-being of their own family / property, so a simple paging vehicle probably isn't sufficient: you want a positive response.
You mentioned that your network & branch locations have grown larger and are more dependent on your computers than the last time you went thru this sort of event. Given the current state of end-user technology, if you haven't already done so, someone in IT should conduct a site visit to ensure that their equipment has a corresponding level of care and preparedness to match your systems. Afterall, it will all be for naught if your systems / servers are up and running and their routers and desktops are without power or are damaged.
If it's been 12 years, then there is a good chance that there are differeing levels of expertise managing those end-user devices, some of whom has given little / no thought to backup or recovery. Also, you don't want to be "surprised" by some new desktop / peripheral eqipment that they bought and installed on their own.
Be sure to review all supply chain requirements for your buildings. As some correspondents have noted, be sure that you what supplies that you can or can't expect to get. That includes fuel and power; janitorial service; food / catering; and water for both machines and people.
Once you have ascertained the status of each it will go a long way to validating your plan, or identifying areas of improvement. (For example, if you plan to have people working extended shifts or 24 x whatever, then you'll need food onsite or they will need to leave and then you lose control over them for random periods of time. So you either need a "feeding plan" or a plan that allows them out of the building for extended periods of time.)
Best of luck!
by david
Generally in DRP and BCP, we set plans that are so subjective we lose sight of the fact that there are threats "upstream" and resultant impacts "downstream".
So when the volcano blows, you may be a thousand miles away, but due to your place in someone else's chain [as well as your own] you are at risk.
eg. the volcano blows ash across a major wine growing district, completely destroying all production - how many jobs are lost downstream? how many businesses fail? what export markets are lost to you [and the wine producers] forever?
So don't just consider personal safety or the loss of infrastructure nearest to the volcano. Think of the ripple effect - from your supplier's suppliers, thru your supplier to you and on to your customers, and their customers.
Create, test and review a plan that looks up and down the chain. And respect those that you depend on and vice versa - it will be the best investment you could make.
That volcano that blew half a world away doesn't look so remote after all - does it?
by zthr2000
by zthr2000
The best way to find out what disasters can happen to your bussines is to get together with poeple of different disciplines of your bussines and start a brainstorm session. Disasters aren't always hardware related, think of foodpoisening, epidemic sickness, heavy weatherconditions causing personel not to come to work, stike, not paying salary.........
By getting the different disciplines togetter wil give you a pretty good idea of what could go wrong.
by kirk bell
by routerqueen
by net-engr
I agree with Kirk and want to expand on his comments.
I read an article a while ago about someone recovering from a disaster. In it, the author thought he and his co-workers had everything covered. Turns out, an important portion of the company used a spreadsheet stored on a Novell file server that hadn't been brought back after the disaster (it was deemed low priority). The business nearly ground to a halt.
It is important to get higher management involved, all the way to the the top. You must identify and *prioritize* ALL business processes (which ones must work first after the disaster, second ...). Then, you have to work with people to document the phone numbers/lines, computer systems (desktops and servers), files, printers, etc. that you must have in place to make each business process work. It's amazing how much you can get done with paper copies of forms, a phone, and something to write with (all important parts of the first and second stages of recovery). Then you need somewhere to store the paperwork... Finally, your company needs to PRACTICE working under the recovery conditions. Hospitals do this regularly when patient information systems are taken down for maintenance. It's amazing how practice will reveal missing links and make everyone more understanding when disaster does strike.
It should be the responsibility of others in the company to address items such as potable water, critical non-IT equipment, etc. BUT, all that said, the comment earlier about the company making sure their employees' families are safe and taken care of is the first priority. If your family is in danger or unable to take care of itself, you are NOT going to be focused on the company!
Good luck!
by zthr2000
by goodoldguy
I was working for an AE firm doing work on the WPPS project when Mt. St. Helens blew. (We were doing off-site DP from the start; someone else has probably done a transition from local to remote processing that could advise.) The ejecta was primarily ash; lava would cause different problems. The powder-fine, gritty ash was a problem for at least a year after the eruption.
You should lose your floppies and drives as soon as you can. The grit got between the heads and the media, and destroyed both. CDs weren't around then. They may be susceptible to damage, and they are at least going to get dirty optics.
Hard drives have a filtered opening to allow for thermal expansion of the air inside. Here, finally, is an objective answer to the 'do I leave my PC on at night?' question: leave it on to minimize what gets through that filter when you turn it off and the air inside the drive cools and contracts. Your hard drive failure rate will still increase, though.
The most important thing for your equipment room is to have positive pressure inside. That is, you want to have a filtration system that draws air from outside the room through a filter, and blows the clean air into the room. A recirculating HVAC system is guaranteed to let dirt in anyplace there's a crack.
Don't buy a new car soon. After the eruption, blow your engine air filter out with an air hose daily, and replace it once a week until cars don't kick up the dust on the roads any more.
Investment strategy: land upwind from the volcano (west of it, typically) will suddenly become more valuable soon.
by gsquared
Having been through the Mt. St. Helen's eruption, I have an idea of what you will go through if you have a major ash-type eruption.
First: Make sure the workplace will be safe and that you have evacuation plans. Get with your local fire department and sheriff. They will appreciate you asking them for plans.
No matter how much you plan for saving the business operations, none of that will be worth doing if the people at your workplace, yourself included, are at risk for health or life.
Dust masks, first aid kits, goggles, clean water, flashlights and spare batteries and bulbs, access to a reliable vehicle that has at least some off-road capability, a couple of good radios and spare batteries for those. Have something to eat and some bottled water. You may be stuck at work for a day if roads are messed up, and going without food or clean water during that time is much worse than going without a secure datacenter.
Make sure employees have communication plans to get ahold of family and for family to get ahold of them.
Second: Once you have human safety planned for and reasonable measures in place, plan for safety of the facility. Are the fire extinguishers certified? Do you have a hose and a water supply for it (you may need to wash ash off the driveway before you can leave)? Air filters and such are necessary, of course. Check with the fire department for further ideas on this subject.
Third: Data safety. Make sure you have offsite backups. If possible, arrange to have a remote host take over any e-commerce databases and applications you have. If all you have to do to continue to take orders and such is re-point your web addresses at a different IP address, you're in great shape.
Fourth: Continuity of business. Even if you can't fulfil orders or deliver services (whatever the business you are in does) during the disaster, you need to have plans in place to get back up and running as rapidly as possible.
Get with management, work out plans based on the idea that your main facility won't have power for 1 day, then plans for no power for 1 week. Most likely, the power will be back some time between those two. Less than 1 day of outage won't cause any major problems for most businesses. Again, the emphasis needs to be on the safety of the people involved and then on the business after they are okay.
If your business involves a lot of customer contact, can you set up a remote location that customer service people, even just a skeleton crew, could go to and take phone calls? That's outside the budget and capacity of most small businesses, but I can't tell from your post how large a business you are in.
In setting up a remote location, the key is going to be getting people there. If highways and such are shut down for a few days, that might be difficult. What you might need to do is contact a call center type business and set up a contingency plan with them. That will only be applicable if you can re-direct phone calls to that site.
Depending on what you do, business-wise, it may be possible to keep operating with something like that.
Also, do you do business locally, regionally, nationally or internationally? If the business is all local, then continuity while the worst is going on is a moot point, since your customers will be in the same boat you're in. Regionally would have a similar effect and at least people would forgive a few lapses if they know you're buried in six feet of volcanic ash. National and international business means some of your customers won't even know it blew up, much less have any sympathy for you. But in that case, you might be big enough to budget for a remote business location, even if it's just a hotel room in a different state and a laptop to connect to your backup host.
The key is prioritizing and planning for contingencies. Start with talking to your fire department and just go from there.
by rdugger
Check out the Emergency Preparedness Guide for Business and Industry at:
http://www.redcross.org/services/disaster/0,1082,0_606_,00.html
Also Institute for Business and Home Safety have a practical guide.:
http://www.ibhs.org/business_protection/
good luck.
-Roy Dugger, Emergency Services Specialist.
by wrap2tyt
As a retired veteran of the US Army, I was stationed on Okinawa, Ja and spent a lot of days away deployed to the PI doing disater recovery. Now that I'm a Security Engineer working in the private industry, I see why disaters take the toll they do, in lives lost and business costs. I can't give you anything new that these good folks here haven't already said, but I will tell you this, empress upon your managers to find out what the employess may need to ride out any disaster situation and do what you can. Look for volunteers to report to work, if the business has a requirement to operate. I wish you luck, I know it's not much of a help but if you need documentation in the form of policies, give a shout with what you need and I'll give you what I got.
donald.jackson@carramerica.com
by chasster
Anyone experiencing a disaster (tornado, monsoon, hurricane, etc.) is open for an experience. Those of us whom have weathered disasters (internationally) have I think, a broader grasp of what it takes to really get into the recovery mode when overseas.
Here in the US we have resources to call upon like the Red Cross, National Guard and beyond. These resources are prepared to walk into a devastated area and perhaps harms way to help survivors. When you are in a foreign country there are many unforeseen issues like: are you welcome in this country, is your visa about to expire, have you had your Malaria & Typhoid shots lately and beyond.
When overseas: if you think the local beer brewery is going to send tankers of fresh water, you?re dreaming.
When overseas: if your waiting on helicopters to evacuate the most critically injured, you?re dreaming.
When overseas: if you think that there is safety in being around the local police in all cases, you?re dreaming.
When overseas: if you are surprised to see local folks retrieving discarded items from the trash so that they can use it in some way to build a new living area, your naive.
When overseas: if your shocked to find a human body discarded (quartered) in the trashcan as this was last nights burglar, you might be next.
A disaster is a disaster, yet without an array of resources to rely upon you may just be the next victim.
Been there, Done that and it is not a pretty site.
E. Charles Sterling
bcdrservices@hal-pc.org
by anaranjoc
Authorities usually have a ma of the most affected areas in case of a volcano eruption, I live in Quito Ecuador, and we planned everyhing carefully, when the plan was ready it was too late, during a weekend the ashes from the volcano covered the whole city, authorities asked people to stay at home you can not go to the office and see what happened, ashes stay in the air for a long time and this caused lots of lightstorms, which obviosly if ignored can cause problems to your equipment. Airports were also closed , ashes can cause damage to jet turbines, so it was not very easy to travel somewhere else. Be sure to have some fans in place , maybe your airconditioning will fail and your computers will be hot.
http://www.ecuador-travel.net/information.volcano.reventador.htm
We have had several other events like this , bu we are more prepared now. People cover their PCs before leaving the office, we have a satellite phone and we have setup an small off location.
Also consider what to do if people is in the office at the moment of the eruption.
Cheers,
Alfred
by chasster
Your experience is True and Real.
When making these plans one must do more than think "outside of the box" particularly if they do not have field experience to draw upon.
One must think of each step as if it were the only step and was a priority one issue to deal with in the minimum of time.
In the real world, One Step in Front of the Devil.
cheers,
c. sterling
PS - Stay alert and alive!
by dbennett
All of the ideas are good, BUT you need to think outside of the box on this one. What good will a generator do you when it will have the same problems as the local power company (ash will clog the intakes).
Do the normal things, like make copies of backups and send them out of the area, look for a hot-site outside the area where business can be continued (if it needs to be), things like.
But in your case, do a few other things for when erupts: have an inside room that has only one door available with some computers wrapped in plastic (a sort of clean room). Also, have plastic ready to cover the doors. This will allow you to grab computers and put them in there for protection (the ash will damage hard drives).
This room can be used to transfer data from the machines to a backup source in a clean environment. You should also have a battery system in this room to run the computers for a limited time (just long enough to get the data off).
Hope this helps a little,
by derithc
We did not have computers in the house when Mt. St. Helen's dusted us with inches and inches of super-fine volcanic ash or 'tephra.' We wore ash masks to keep from breathing too much of it, but we still had to blow our noses and cough it out. Plastic on the insides of the windows slowed down the infiltration, but it didn't keep it all out and we'd have to replace the tape because the ash would collect under it and loosen it. Also, we had to enter and leave the rooms and buldings. We lost cassette players, turn tables, cars' air fliters, paint was both scratched and etched. Everything that had moving parts got the grit in between them. And, while the worst of it was the year the mountain blew, it wasn't over when the rain came. We had ash dust us for several years after. It would dry into a hard crust then the wind would carry the extremely fine dust back into the buildings.
You will need a room that is so filtered and so air tight that nothing gets in right from the street, including you. Taping up plastic won't be sufficient. I suggest tight seals on the door. Nothing with key holes. Windows, if you have to have them, that don't open. And HVAC (heating, ventillation and air conditioning) designed to filter out the smallest particulate.
Have you considered going underground to house all the equipment and back up energy? The banks did.
Here are some links you might find interesting:
http://www.nwp.usace.army.mil/op/b/msh.htm
http://dominicapsn.freeyellow.com/Hazards/Tephra.html
Good luck!
Derith
by yanipen
I live here in the Philippines, particularly in Zambales where Mt. Pinatubo is. When it erupted, it covered a whole lot of hundreds of square kilometers of ash and sand. This is due to the fact that when it exploded, there was a huge storm present. The storm helped the ash and sand to reach a large area. I mean, a large area.
Have you seen Dante's Peak? That movie is very close to what happened here.
All of the guys who gave their point of view before me are all correct. The fuel for the generators, the backups whether in-site or off-site. But none of them mentioned this: Transportation. Logistics. Need I say more?
Of course, when all of the planning about disaster recovery have been excuted, how come nobody thought about getting your equipments and yourself out?
by clindell
by warpindy
by lupa
This discussion is closed to new replies.
Trending IT Employment posts
Sort by Trending