We've all done it -- made that stupid mistake and hoped nobody saw it, prayed that it wouldn't have an adverse effect on the systems or the network. And it's usually okay, so long as the mistake didn't happen in the data center. It's one thing to let your inner knucklehead come out around end user desktop machines. But when you're in the server room, that knucklehead needs to be kept in check. Whether you're setting up the data center or managing it, you must always use the utmost caution.
Well, you know what they say about the best laid plans... Eventually you will slip up. But knowing about some of the more common mistakes can help you avoid them.
1: Cable gaffes
You know the old adage -- measure twice, cut once. How many times have you visited a data center to see cables everywhere? On the floor, hanging down from drop ceilings, looped over server racks and over desks. This should simply not happen. Cable layout should be given the care it needs. Not only is it a safety hazard, it is also a disaster waiting to happen. Someone gets tangled up and goes down -- you run the risk of a law suit AND data loss, all because someone was too lazy to measure cable runs or take the time to zip tie some Cat5.
2: Drink disasters
I know, this might seem crazy, but I've witnessed it first hand too many times. Admins (or other IT staff) enter the data center, drink in hand, and spill that drink onto (or into) a piece of equipment. In a split second, that equipment goes from life to death with no chance for you to save it. Every data center should have a highly visible sign that says, "No drink or food allowed. Period." This policy must be enforced with zero tolerance or exception. Even covered drinks should be banned.
3: Electricity failures
This applies to nearly any electricity problem: accidentally shutting off power, lack of battery backups, no generator, pulling too much power from a single source. Electricity in the data center is your only means of life. Without it, your data center is nothing. At the same time, electricity is your worst enemy. If you do not design your electrical needs in such a way as to prevent failures, your data center begins its life at a disadvantage. Make sure all circuit breakers (and any other switch that could cause an accidental power loss) have covers and that your fire alarms and cutoff switches are not located where they might tempt pranksters.
4: Security blunders
How many keys to your data center have you given out? Do you have a spreadsheet with every name associated with every key? If not, why? If you aren't keeping track of who has access to the data center, you might as well open up the door and say, "Come steal my data!" And what about that time you propped the exit door open so you could carry in all of those blades and cable? How much time was that open door left unattended? Or what about when you gave out the security code to the intern or the delivery man to make your job easier.... See where this is going?
5: Pigpen foibles
When you step into data center, what is your first impression? Would you bring the CEO of the company into that data center and say, "This is the empire your money has paid for?" Or would you need a day's notice before letting the chairman of the board lay eyes on your work?
6: Documentation dereliction
How exactly did you map out that network? What are the domain credentials and which server does what? If you're about to head out for vacation, and you've neglected to document your data center, your second in command might have a bit of drama on his or her hands. Or worse, even you've forgotten the domain admin credentials. I know, I know -- fat chance. But there's this guy named Murphy. He has this law. You know how it goes. If you're not documenting your data center, eventually the fates will decide it's time to deal you a dirty hand and you will have a tangled mess to sift through.
7: Desktop fun
How many times have you caught yourself or IT staff using one of the machines in the data center as a desktop? Unless that machine is a Linux or Mac desktop, one time is all it takes to send something like the sexy.exe virus running rampant through your data center. Yes, an end user can do the same thing. But why risk having that problem originate in the heart of your network topology? Sure, it'd be cool to host a LAN party in your data center and invite all your buds for a round of CoD or WoW. Just don't.
8: Forgotten commitments
When was the last time you actually visited your data center? Or did you just "set it and forget it"? Do you think that because you can remote into your data center everything is okay? Shame on you. That data center needs a regular visit. It doesn't need to be an all-day tour. Just stop by to check batteries, temperature, cabling, etc. If you fail to give the data center the face time it needs, you could wind up with a disaster on your hands.
9: Tourist traps
You're proud of your data center -- so much so, you want to show it off to the outside world. So you bring in the press; you allow tours to walk through and take in its utter awesomeness. But then one of those tourists gets a bit too curious and down goes the network. You've spent hundreds of thousands of dollars on that data center (or maybe just tens of thousands --- or even just thousands). You can't risk the prying eyes and fingers of the public to gain access to the tenth wonder of the world.
10: Midnight massacre
Don't deny it: You've spent all-nighters locked in your data center. Whether it was a server rebuild or a downed data network, you've sucked down enough caffeine that you're absolutely sure you're awake enough to do your job and do it right. Famous. Last. Words. If you've already spent nine or 10 hours at work, the last thing you need to do is spend another five or 10 trying to fix something. Most likely you'll break more things than you fix. If you have third-shift staff members, let them take care of the problem. Or solve the issue in shifts. Don't try to be a hero and lock yourself in the data center for "however long it takes." Be smart.
Have you ever witnessed (or been part of) a data center disaster triggered by a silly mistake? Share your experiences with fellow TechRepublic members.
- The 21st Century Data Center (ZDNet special report page)
- Executive Guide: The 21st century data center (free ebook)
Jack Wallen is an award-winning writer for TechRepublic and Linux.com. He’s an avid promoter of open source and the voice of The Android Expert. For more news about Jack Wallen, visit his website getjackd.net.