Discussion Locked

Scheduled Building Power Outage

0 helpful
25 replies
Closed to new repliesThis discussion is closed. It stays readable, but it takes no new replies.
I a couple of weeks the building we are in is having to do some repairs to the electrical system. Our power will be shut down for at least 12 hours. I plan on shutting down the servers just before they start. But management has come to me saying they want our email to still be flowing since 90% of our business is done via email. Now I was thinking of moving there server off site but that means alot of configuration changes, right? Are there any other alternatives out there that I can explore? Or are they out of luck during this time? Thanks.

25 replies

by jamesrl

Just the act of unplugging them, and setting them up in another domain, re-iping them, and then of course doing it all over again - that might take most of 6 hours anyway - perhaps an exageration. But just think how long it takes for a new IP to propagate.....

Take the email down at the quietest time in the day before the shutdown. Most systems will try to keep sending mail for some time. I would also suggest putting up an auto response for a week before hand notifying people sending mail to you of the outage.

Perhaps there is some service that would allow you to have another mail system take your messages for some time? Not sure how it would work.

James

by faradhi

When a local hospital had a fire in their electrical room they had to pay to have a high powered generator put in an alley. They ran a cable up 9 stories to power the data center. Of course, they had to have 5-9s of uptime and already had this worked out.

Something similar may be in order for your situation if management is requiring the email to be up. However, I have found that when the price hits management, they may find the downtime acceptable.

by jaqui

are you sure it wouldn't be $XXX,XXX.xx?
[ the tech to run the danged genny costs extra, and is required if you don't have someone in house with the right piece of paper. Then the city might demand a permit for having it sitting out in the alley / loading dock and running. ]

You are right in that it's the only way to keep email servers online during the scheduled power outage.

by cactus pete

Postini, among other service providers, will be your MX record, and will take delivery of your email before handing it to your own systems. If your systems are unreachable, you can still get to the postini site and check your email. They also filter for viruses and spam, etc...

When your systems come up again, they start to feed the mail back to you.

We don't use them, but we looked into it.

We have a secondary site (several, actually) and the secondary site acts as the secondary MX record, queuing up the emails until they can be handed off to the appropriate servers.

by Member 30634329

We find that not only is Postini a great service for filtering spam and viruses before they hit your corporate network, but it also works great during the occasional outage by holding email in a queue until systems are restored.

by slw171

Thanks to all of you who responded. It looks they are going to have to be without email for a while then. I did not think there was another option but just wanted to ask to be sure. We use Appriver for our spam and virus protection. So I know all the email will be held until the server comes back online. Thanks again to everyone you were all very helpful.

by ldemetse

This is going to sound really archaic, but it works! We rented a generator for just such a situation and it worked. We were able to plug our UPS in to the generator, powering both the mail server and the communication equipment necessary to send/receive emails.

by buzzwang

Does your company have a large UPS or a bank of UPS's? You could try cycling through those if they'll provide enough runtime.

Also, we're doing the same kinda of thing in Feburary in prep for a generator installation. We're having the guys doing the install provide a smaller generator that they are going to wire in directly to our server room via the main AC line that runs to the UPS units. We'll run on UPS for about 35 ro 40 minutes so they can wire in the generator, and then we'll run on that until they finish up the rewire at our utility company access. Then we switch back to what will be fully charged UPS's while thye pull off the temp generator and restore the wiring.

You could also rent a portable generator (if you can find one with the capacity you need) and run a heavy duty extension cord and outlet box into the building from outside to run that server, roouters, and other essential equipment. You'll have to top off the fuel take every 3 or 4 hours, but it's a lot cheaper than most other options.

by ame4c

You should find an alternate power supply, either UPS or generator. We have both here. The UPS lasts about 2 hours. However, for your situation I would consider the rental generator. You will probably find it more cost efficent.

by slw171

yes we do have a bank of UPS but not a large one. The generator idea is a good one but for one we are in a large high rise building. Plus the management company here is pretty rigid on that stuff. But thanks for the response

by tonybaggadonuts

You wouldn't have to worry about the whole building, you can just plug your USBs, servers, switches and whatnot into the generator. Are you the high-rise building, or is your company on a couple/few floors like mine? If that happened here, I would just plug my exchange server into a generator and the nessesary switches. Phone and internet would be online?

All your looking to keep up is email, right? is that one server? even easier.

If management isn't willing to pay a few hundred clams to keep the email running for 12 hours, so they can keep their ebusiness going, then maybe they need some persuading. do a cost analysis of what the Co. would loose if no email for 12 hrs and what it would cost for the generator rental.

by jleather

Have you asked your electric company for suggestions or if they have had companies with a similar issue and how they handled it? (And perhaps they can give you a contact). Also the company doing the electric repairs (if different from the electric company) may also have seen this before.

A generator is the only thing I can think of myself, but what have you got to lose to make a few phone calls to the above companies?

Good luck . . .

by kellylists

If you secure an alternative source of power (generator, batteries, etc.) don't forget to provide power to data circuit equipment in the DEMARC room, any MUXes or relays bringing the circuit to your office, the Internet router, and firewall servers.

As a former ISP owner and user of Postini, if your management truely feels the need access email during the outage, I can vouch for Postini being a viable option. But ask them where they are going to be located during the outage, if they would need access to your other systems to handle customer inquires and you may find that they need all the systems to process customer emails.

My guess is they will decide they can survive a 12-hour outage without major damage given the alternative costs.

Kelly Breit
Business Continuity/Disaster Recovery Consultant
kelly@breit.com

by slw171

I forgot all about that until the other day. Thanks for bringing that up. I am pushing hard for them to just be down for 12 hours and to start sending emails to our customers about the outage. Management advised me they are getting tight with the budget so I think they are going to have to live without.

by ppowell

Most properly configured mail servers will hold your mail for delivery (generally 1 -2 days)and retry delivery periodically if your server is unavailable for a period.

You should not lose any messages being delivered.

by t.matthies

You are fortunate to have the foreknowledge of this outage, but what if it was unplanned and you didn't know how long it would continue?
Ask management what a day without their systems would cost the business and compare it to the cost of some contingency planning and disaster recovery planning. They may realise the value of planning now for future possible events that are outside of their control.

by hawad1001

Check with your ISP to see if they can cache your email during the outage. We use Verizon (formerly MCI/UUNET) and they have a free email caching service for their customers. But you will need to notify the ISP ahead of time. Moving your server is too time consuming and would require a lot of work since you would have to change your DNS/MX records and have it propagate everytime you change the IP address of the server.

by itilfriend

Sounds like you have received some great info to help with this power outage, just don't underestimate the time each end of the power outage for shutdown and startup of your systems.
We had something similar in June '05 where the power was out for 12hrs also, but we ended up scheduling close to a 24hr offline period to cope with a graduated shutdown and startup.
Just think carefully about what goes down and up when and what can be done earlier (dev and test envs) and brought up later than the others to shorten the outage for your production systems.
Enjoy!

by cactus pete

I think Postini will give you a trial run for 30 days. You might be able to use that... Then show it to your bosses as a good solution to keep.

by sean.rackley

When you say building, the whole thing? Including workstations? Or just the server room? If the whole building how do management and users expect to power there PC's to get there email?

by a contractor

If this is going to be an ongoing issue ( the power supply and or the building management) time to move your business critical servers to a hosted environment company. They can make sure those kinds of things are VERY low risk situations.

by neoconone

In terms of time and lost productivity, by far the cheapest and easiest solution is to keep the infastructure in place and simply provide an alternative power supply. Are there any logistical reasons why you cannot rent a small generator, plug your UPSs into it (for clean power to your server(s)), and drive on? You need never lose any up-time, as the UPSs would be unplugged just long enough to plug them first into the generator, and again back into the wall outlet once power is restored. Should cost under $300 for rental, gas, long 10 AWG extension cord, and chain and lock for the generator. And a fraction of the time of doing something like moving your servers offsite and going through the config/reconfig exercise. Just a thought.

by omalleyk3

Rent a generator to power the whole building. Everything will continue to function. It will be a great experience for you if you ever have a natural disaster to deal with. What would you do if the power was out for 2 weeks?

This discussion is closed to new replies.