Being a fairly new administrator of a small datacenter of 15 servers running through a 4500 series core switch, 7 other 3750 switches and a host of perimeter firewalls, anti-spam, RAS units etc...I had been tasked with testing the software component of our UPS.

Power Chute runs on select servers, and when main power is cut the Symmetra tells Power Chute to perform graceful shutdowns on select systems to extend the Symmetra's battery life. To test this I opened a PowerChute application which communicated with the Symmetra; and ordered it to perform a 'self-test', believing that the Symmetra would simulate a power failure and order a graceful shutdown of selected servers.


Right when I was marvelling in how effective the servers had been dynamically shutdown, every-and I mean every-single little green light in the entire server room went out. Every light, every fan, every server, router, firewall, switch and appliance...including the Xiotech 2TB SAN. Everything simultaneously went out in an instant.

What happened? Here's where it gets embarrassing. Not only did the Symmetra self-test initiate software shutdowns of the selected servers, it also peformed a shut down of the UPS itself-which ought not be done when such a UPS is in production with several hundred thousand dollars of assets running on it!

The nature of the fiber optic SAN and the Cisco environment meant that it would take about 2 to 3 hours to get everything up again. That's a lot of time for a lot of Homer Simpson "Doh!" exclaimations