Discussion Locked

Disaster Recovery - Hurricane

0 helpful
122 replies
Closed to new repliesThis discussion is closed. It stays readable, but it takes no new replies.

This is page 2 of the discussion. Read the opening post

122 replies

by hal 9000

The water involved here was not clean fresh or sea water it contained contaminates initially and with the water inundation would have picked up more contaminates so you should treat these objects as Biological Hazards and act accordingly. Use Surgical Gloves when handling any just fished out of the water hardware and think about this there was a Hurricane so in most cases there would have been no power available when the water hit even if the unit was plugged in and turned on it would have stopped some time previously to the water intrusion.

When we where cleaning up after the AU Day floods in 74 we discovered things ranging from bottles of medical O2 to aircraft wings and everything in between all of this was trashed and thrown away as no one wanted the recovered items and in the case of the aircraft wings they would have been condemned by the Civil Aviation Authority because of the possibility of Corrosion and resulting failure. About the only thing that I ever heard about that was used by the finders was some wax candles. In most of the areas affected there was not just water but fast flowing water involved so we had a real mess to deal with and quite a lot of the mainframe rooms didn't get pumped out until we had someone there to start immediately on the hardware.

It's a really interesting experience using a fire hose to clean out a Main Frame room with several Million $ worth of hardware involved, unless you have experienced it you will never know the feeling that it generates. ;)

But even after the clean out several years latter when we where ripping up cables we still found hardened muck in the cable trenches and we used enough water in the clean out to fill the rooms several times over but as we hosed it out the water was being pumped along with the dislodged sludge. We even hit the old HDD with the removable platters with the fire hose after we had removed any platter elements in the box and literally allowed it to fill up and overflow until the water was coming out clean. For the Tape Drive Banks we used the same method but with far less pressure and a big rubber type wedgie to push the muck to the pumping point so it could eventually be sucked out of the place. :p

While it was fun at the time it is something that I hope I never see again. The units that where rescued and left to dry before we could get near them where the ones that where thrown out as they had been destroyed after only 3 days there was a massive amount of corrosion on the circuit boards to make any attempt at repair not worth while. The only good thing about this flood was it was only about 30 feet of water through the main business district so all the Backup Data could stay on-site in the higher rooms of the buildings and remain unaffected so after we replaced the hardware we could have the place up and running in very short order. :^O

I still remember the smell and the Hep shots that we all had to have not something that I would recommend to the average tech. :D

But if you are going to be working on this stuff you will have to remember to treat is all as Biological Hazardous and act accordingly and have whatever shots are recommended by the Doctors, Booster shots for Tetanus and Hep are the bare minimum and extremely important even if they do hurt like hell for weeks after the event. The Hep shots consist of 3 shots at 2 week intervals and are not cheap so if you are working on this stuff you should be getting your shots now and have them paid for by the Boss so you are covered against the worst type of disease.

Col ]:)

by breadtrk

FEMA? Sorry guy but I have to laugh my arse off at that one. There are many full time pro's posting here that equipment CAN be saved and you put all your baskets in a guy from FEMA.

Since your position is at a fairly important place in the community, you probably have insurance to replace everything so the guy from FEMA has no relevance in this discussion.

BTW, still laughing.

by angry_white_male

Most hardware damaged by a hurricane from flooding (especially a storm surge or catastrophic flood) is going to be written off as a loss.

By the time you can get to the equipment, most of the salt water that has receded back to the ocean had dried and everything inside the case is toast. The residual salt and corrosion from the heat/humidity down there alone is enough to kill it. Also be very wary of mold contamination - so when you go into retrieve your hardware, be sure to take the proper precautions (mold infestations can cause serious health complications.

For those of you in New Orleans, the flood waters there are contaminated with god-knows-what - there will be considerable amounts of silt and other contaminants in the water that will get into anything and everything. Try to recover your tapes and drives, then write off the rest.

This should be a lesson to all of us to remember to back up our data, store the tapes off-site somewhere, keep your data rooms above ground level (in a flood-prone area - the higher, the better)... be sure to take any critical data with you if you are ordered to evacuate (shut everything down, hopefully you have hot-swap drives you can pull and take with you - be sure to label them and keep them in order). If you're not in a flood-zone, keep tarps and bungee cords handy to cover everything up with in case you lose part or all of your roof to keep the equipment from getting soaked. If your server racks have (or had ) doors - use them, protect your equipment from flying debris.

Any large off-site data center that operates in a flood-prone area is asking for trouble. Think about relocating to higher ground in an area that's less prone to natural disaster.

Finally - have insurance that'll cover your equipment losses! Have a disaster recovery / business continuity plan in place that covers these scenarios. The plan should include agreements with your vendors to replace hardware quickly, a place to set up a temporary server room, back-up power, some kind of internet access (DSL, T1, broadband, satellite, whatever), etc. In a Katrina scenario - you can be certain there will be a surge of orders to replace equipment, expect your stuff to be on back-order, so expect some delays. Data center trailers designed for disaster recovery efforts will be extremely scarce. Make sure your customer service people, account reps, PR, etc... are in the loop so they can keep customers in the loop with what's going on. Plan plan plan!!

The questions I'm seeing here shouldn't be "how do I make my submerged server work again". I sympathize with each and everyone one of you affected by Katrina, but if you had a plan in place - there would be less headaches to contend with in the recovery effort. The questions here should turn to "how do I prepare for a disaster of this magnitude and what can I do to minimize my losses"??

Just my 2 cents.

by warwizard

I addressed the corrosion on the CMOS battery sockets. If you're talking about the system board for a 4 way or 8 way server, the cost may exceed $2,000 for the replacement board, and you might have 10 of these systems setting there. a 40% recovery rate saves $8,000, and gets 4 servers up till replacements can be procured. Just getting 1 server up provides a platform to restore your backups to prior to sending the data to the replacement systems, so there could be good reasons to recover some of the hardware.
Recovering the primary domain controller and schema master could save you weeks of work.

warwizard

by mt_abbas

What is the best way to hunt a job with companies engaged in recovery of IT Assets post Katrina. Any names of companies would be welcome

by jcrobso

Last year I left a flash drive in my pants pocket.
My wife did the laundary and washed and dried it!!
I let the flashdrive sit for a few days to make sure all the water was gone and it's working great. John

by wayne.walker

I've dunked several Radio Control recievers, servos & speed controllers in salt water.

You can sometimes clean them as noted above, but eventually the small wires & other parts succumb to corrosion and fail. It is a lot of effort and generally not worth the effort.

The only exception to the above is HDDs with valuable info on them, we send failed drives out to MDS Disk restore, 11750 Stirling Ave.
Riverside, CA 92503, Gary (951) 352-2425

They're the best!

by dean

Interesting topic. The best I have done for a soaked computer was like someone above said. Pull it to bits and clean it with water, then let it dry. Although I didn't put mine in the oven. I dunno how warm 125F is. lol. But i put it in the cupboard with the hot water cylinder for a week to dry it out. Just hope you dont get small wires rusting badly, Hopefully it wasn't wet for too long.

by ksmclean

It's worth a try - a client of mine had their facilities swamped by floodwaters a few years back. We dismantled all the PCs, washed the boards with warm soapy water, dried them using a blowdryer then reassembled everything. 97% of the computers worked on first powerup. The rest just needed 1-2 boards replaced.

Good luck!

by jhkotak

My experience is based on the flooding due to heave rains in Mumbai.I had visited one of the enginnering colleges and they were doing the following things to recover from the water daamged computers.

1. Do not try to switch on the computers and immediately disconnect from the power supply.

2. Open the cabinet and dissamble the components as much as possible, ie power supply, mother board, cards - etc display, modem, lan

3. Clean up wipe off with dry cloth as much water, dirt as possible.

4. Dry the components using one of the following methods
a. Expose to high watt, high heat radiating tungsten lamps for quite a long time - say 2 days.
b. if above not possible/available then dry with say hair dryer( see if you can get industrial grade and styling hair dryers will have to switched off every 15 min )
b. Clean up with alcohol based solution and wait to dry again.

5. Switch on the SMPS and check with multimeter if the volatge is proper before connecting to motherboard, HDD etc.

6. Re-asemble the computers. Keep your fingers crossed and then plug and pray.

With the above method the engg. institute has been able to recover 80% of their computers.

by pritchards

I used to work in a Factory that made PC boards.
We used to use a Ultra-sonic bath with a cemical called Fre-aunt (well that is how we said it, have no idea how to spell it). We used to submerge whole PC's in it and switch them on a hour after they came out. This is how ever 100% Ozone unfrindly stuff and may not be legal in the US. GOOD LUCK!!!

by bmedlock

Most major data recovery services, if not all of them, are giving discounts to victims of Katrina. Ontrack, DriveSavers, & DTI DATA I know for sure are. You may be able to recover what you need using the advice found here, there's lots of good tips, but before you try anything please evaluate the risks. None of these tips are garaunteed, and trying to recover the data yourself may mean it can't be recovered later. If you have irreplacable data that is not backed up, I'd check into a professional service. Most will evaluate and give you an estimate at no charge if you are a victim of Katrina, and if nothing can be recovered there is no charge.
Not to mention your insurance will probably pay for it. And if you didn't have insurance for this sort of thing the government might help. The point is there's tons of help out there to cover the costs, and you can get it at a discount, so why risk it?

by Member 23452619

DTEC International has authorized the Washington DC Metro Eastern Forensic Office to accept damaged HDD and CD Media from consumers and corporations who suffered loss in the wake of the passing Hurricanes. This program will be announced in our October publication of our Newsletter and on our company web site. To contact DTEC and learn more about qualifying for this Discounted service please contact recovery@datatexcorp.com

Our discount service is designed to cover the cost of the materials we use to execute the recovery. In general we will analyze the media sent, provide a free assessment and probability of recovery, and provide a fee estimate. The normal estimate is between 80-90% off our standard service rates and are designed to assist the victims not take more from them but rather give something back. In most cases consumers will request personal data, pictures, etc...Corporate Customers and Government agencies will be looking for more extensive damage. We are here to help.

Possible financial assistance to those that qualify to assist in the offset of recovery fees.

DTEC has provided Data Recovery and Forensic digital recovery services for the past 11 years. With our network of Recovery professionals world wide and our local offices staffed in the USA we are committed to assisting those who have suffered from this tragic natural event.

Learn more about DTEC at www.datatexcorp.com
This message was posted under the authority of the DTEC Group by a member of DTEC International Upper Management Team.

by alexit

Because the machine had been powered on, creating friction in the platter. This causes expansion, cold in the freezer forces it to shrink, there for makes scratches abrasions smaller. The more time in the freezer the more time you have to then shut down and pull the drive and send it off for recovery.

by macoco

I've always heard that oil is a conductor, so you might as well leave it wet...you'll get the same results...smoke.

by macoco

I think you read too much into this post...but if you're right, they are taking applications wt Halliburton...where have I heard that name before?

by beads

I haven't had to do this in sometime but I used to put keyboards in the dishwasher with minimal to no drying time. Just leave em out to dry for a day or so depending on humidity. No, really. It does work - at least with those older heavy duty keyboards through the older Compaq keyboards.

Now, WD-40 would probably pop every capacitor off the mobo. Remember thier is a difference between electrical and ELECTRONIC - lol. Yeah, its great for old carburators and lawn mowers too! Computers? Ummmm... no.

- beads

by lburner

I have been in computers from the git go, when they had vacuum tubes, and an ET longer than that and, yes, we did recover dunked equipment in the military.

NO WD-40 or other harsh solvents!

The person who said to remove the hard drive and hoser down is right. I would break the whole thing down to the major components.
Then flush with IP, we have always washed the boards with alcohol as we assembled them. It removes resins and solvents you don't want. Let it air dry or blow things out with low air pressure. I would stay away from applied heat unless you know what your doing and can control it.
Do replace the keyboard and mouse. They are not cost effective to mess with if they are failing.

You may have to rock all plugins including some chips, this is where good contact cleaner may be needed.
Let it all set for a few days to thourouly dry out, reassemble and give it a go.
At this point I would use a substitue HD if your hard drive has valuable business data stored on it.

Two parting comments.
You never realize the value of data back-up until you whished you had done it.

Systems are so inexpensive today that it is easier and cheaper to just replace it and install the old hard drive with high hopes that it was well hermetically sealed.

Best bet: distilled water followed with IP and air.

by Member 23452619

DTEC International has authorized the Washington DC Metro Eastern Forensic Office to accept damaged HDD and CD Media from consumers and corporations who suffered loss in the wake of the passing Hurricanes. This program will be announced in our October publication of our Newsletter and on our company web site. To contact DTEC and learn more about qualifying for this Discounted service please contact recovery@datatexcorp.com

Our discount service is designed to cover the cost of the materials we use to execute the recovery. In general we will analyze the media sent, provide a free assessment and probability of recovery, and provide a fee estimate. The normal estimate is between 80-90% off our standard service rates and are designed to assist the victims not take more from them but rather give something back. In most cases consumers will request personal data, pictures, etc...Corporate Customers and Government agencies will be looking for more extensive damage. We are here to help.

Possible financial assistance to those that qualify to assist in the offset of recovery fees.

DTEC has provided Data Recovery and Forensic digital recovery services for the past 11 years. With our network of Recovery professionals world wide and our local offices staffed in the USA we are committed to assisting those who have suffered from this tragic natural event.

Learn more about DTEC at www.datatexcorp.com
This message was posted under the authority of the DTEC Group by a member of DTEC International Upper Management Team.

by Member 23452619

DTEC International has authorized the Washington DC Metro Eastern Forensic Office to accept damaged HDD and CD Media from consumers and corporations who suffered loss in the wake of the passing Hurricanes. This program will be announced in our October publication of our Newsletter and on our company web site. To contact DTEC and learn more about qualifying for this Discounted service please contact recovery@datatexcorp.com

Our discount service is designed to cover the cost of the materials we use to execute the recovery. In general we will analyze the media sent, provide a free assessment and probability of recovery, and provide a fee estimate. The normal estimate is between 80-90% off our standard service rates and are designed to assist the victims not take more from them but rather give something back. In most cases consumers will request personal data, pictures, etc...Corporate Customers and Government agencies will be looking for more extensive damage. We are here to help.

Possible financial assistance to those that qualify to assist in the offset of recovery fees.

DTEC has provided Data Recovery and Forensic digital recovery services for the past 11 years. With our network of Recovery professionals world wide and our local offices staffed in the USA we are committed to assisting those who have suffered from this tragic natural event.

Learn more about DTEC at www.datatexcorp.com
This message was posted under the authority of the DTEC Group by a member of DTEC International Upper Management Team.

by nisquallypauli

WD-40 is great for a quick fix.
But in the long run its a varnish.
It be used in machine shops to protect
from rust on stored equipment.
bad idea.
There are electrical cleaners for the
cards and such.
I survived a flood in 1996. A cleaning did
the job on mother board, cards.
Drives you could only hope water didn't
get inside. Sometimes, sometimes not
I wouldn't fire up any thing that seemed
a bit wet.

Nisq Paul

by cemetary_ragdoll

this is tina.. looking for my father... if you were to put more info on yourself somewhere, i mgiht be able to find you.. if this is you.. seems like you

This discussion is closed to new replies.