General discussion

Locked

Lesson For Other Net Admins

By radiic ·
Yesterday was the typical network administrator duties, until about 2:30 in the afternoon. One of the managers came by my office and said you might want to come take a look at this. I smiled and thought wow she must of did some wild decorations forHolloween or something. She said I dont think youll be smiling long. Bewildered I followed her to my server room. Turns out she had been letting a carpet guy messure the building for new carpet. She has access to the server room and when they walked in they found what was to be my nightmare.

I have seen rain forests on TV, but I never thought in my wildest dreams that I would see one in my server room. Water just pouring through the ceiling tiles. My god....as i looked and assesed whatthe water was pouring on, all I could say was NO F***ing way.

Thank god somebody around had the good sense to allready have gotten some visquene and was on the way with it. When they showed up, I was startled but started into action. While I was trying to divert the water away from flowing over the top of my catalyst switch and routers and servers, that nightmarish thought that all network administrators fear popped into my head. How in the **** am I going to fix all this equipment and get it back running. I have a DRP. Don't think waterfalls is covered in it. (mental note) add severe water falls to DRP. And would I actually take the time to read it before I started into the task of trying to get equipment back up and running, nah...But it is good that I have one on the shelf in my office.

cont...

This conversation is currently closed to new comments.

5 total posts (Page 1 of 1)  
| Thread display: Collapse - | Expand +

All Comments

Collapse -

Part Two

by radiic In reply to Lesson For Other Net Admi ...

Turns out that there is an airconditioner in the server room right above the rack for my equipment for this building. There was a freon leak and a big chunk of ICE had developed the building manager and the air conditioner guy didnt think that it was a big deal and decided to turn of the air flowing in that unit and let the ice melt. Well something happened to the drainage pan and lucky me started flowing over my equipment. Wild thing is that It had flowed over some of my network cabling in the ceiling and let a nice stead flow directly to the patch panel. It stopped afater a while. I was able to shut down a couple of my primary servers before the monitor burnt out. I was able to Unplug several switches and routers. And I used some canned air to **** water away and some of the plugs. I left every thing off overnight.

I came in this morning replaced the monitor and started booting my servers...I have never heard a noise so lovely as that of SCSI drives spinning up. Then I crossedmy fingers and plugged in the catalyst...Whir...Whir...Whir...I plugged in my console cable really quick and was shocked to see it booting. Routers up. Even my PIX. Needless to say I am sitting here writting this now.

My question is this...What would you have done. I am now sitting here wondering if I just got lucky as **** ( and should go buy some lotto tickets) or if I really did the right thing. And if I did the right thing I should add it to my DRP.

I inheritted this server room and had no clue that there was an air-conditioner above the racks but you can bet I will be checking all my server rooms and designing some sort of catch and drain system above my racks.

Rad

Collapse -

by tbragsda In reply to Part Two

Lucky, yes.

I had a flood once, just to a client area not server room. Most of the equipment came back OK, but I would be on my way to LV right now if I were you.

Realy, glad to know it worked out OK. Started to read the post, and was feeling bad for the weekend you had coming. Have your boss take you out for a nice soothing drink tonight, come in this weekend, check it all out, repair any remaining problems, and GET THE HVAC GUYS TO MOVE THAT AC.

Collapse -

Good job, But!!!!!!!

by Fvazquez In reply to Part Two

I think in my opinion that you handled the situation very good. But I also think that every DRP should include any possible and almost impossible situation that will happened in a systems DRP. That will include disasters like Fire, flood (doesn?t matter if you live in a area that this thin doesn?t occurred often or simply never happens) and in these day we even have to think in a terrorisms situation just to be reasonable prepared for any possible event may happen. Atmospherically disaster can happen any time without any advice. Earthquakes, hurricanes and tornados may affect any operation at any time.

The big lesson here is BE PREPARED FOR THE POSSIBLE AND THE IMPOSSIBLE. Maybe some people will think that you are ridiculous, but as I said IS MY A$$ not their that is on the edge. I live in the Caribbean so we are aware of all this events in ours DRP.

Collapse -

Great work !! Hats off to you!

by Oz_Media In reply to Good job, But!!!!!!!

Sounds like you possess a strong sense of logic and are able to resolve issues without too much panic (OK maybe just a little, right?).
I know I would have gone for the canned air and probably a zillion KimWipes.
It sounds like you are aware ofand will update your DRP, what more can your employer ask for.
Pat yourselfon the back (then smoke a fatty to calm down a bit) ;-)

Collapse -

Ouch!

by admin In reply to Part Two

I don't know what you would have done differently, But I did go up in the attic and check out our HVAC etc. over my areas. The worst thing is that sometimes the damage with water in electronics isn't immediately apparant. Thanks for sharing. We are setting up a second server room in our main location and checking the buildings pipes, vents etc. is now on my list of things to do!

:)

Back to IT Employment Forum
5 total posts (Page 1 of 1)  

Related Discussions

Related Forums