Discussion Locked

Disaster Recovery? Waste of money, we've been fine the last few years...

0 helpful
9 replies
Closed to new repliesThis discussion is closed. It stays readable, but it takes no new replies.

Yeah... the title represents what my CEO thinks of Disaster Recovery.
Quick Background: My boss, the Director of IT... just quit one day.
Since he quit, I have researched hours upon hours into his undocumented infrastructure he created and surprisingly enough... I have managed to get a network profile together in 4 weeks with the help of consultants.

Now. I suggested a Disaster Recovery plan that included: (Backup Exec 11)
OFFSITE SERVER BACKUPS in coldstate (plans to develop a failover system next)

1) D2D (disk2Disk) system
2) CD/DVD secondary backup
3) Tape Archive

1) Full Backup everynight
2) Snapshots every 10 minutes during business hours
3) Saturday recovery of Business Week's full Backup. (peace of mind to know my backup works...)
4. Multiple site storage for CD/DVD and Tape Archives.

My plan included putting my onsite servers on a generator in case of powerloss, or at the very least, upgrading the $199 UPS specials my ex-boss thought was sufficient to protect our annual $60 million revenue...

Right now... my companys disaster recovery is ummm...
an incremental backup via Veritas 9.1 on each server utilizing ONLY TAPE DRIVES stored on-site.
Yeah... how do I convince them that this is needed... before something happens due to shortsightedness?
Please Comment :)

9 replies

by charliespencer

1) Your boss is planning on nothing ever happening: no fire, no flood, no meteor through the data center. He's willing to risk long-term service loss to save money. In short, he's an idiot.

2) You posted this late on a Friday, so you probably didn't get any response over the weekend. Hopefully this comment will bump it back up the list.

by gadgetgirl

Locked a few of the "higher ups" in the organisation in a room off-site for an entire afternoon. Made them do a table-top DR exercise - it was the only way I could get them to understand just how critical IT was for the business. Every hour was classed as a business day, so at 1pm it was Monday, 2pm it was Tuesday etc.

I roped in the chairman, as it was the only way to get some of the directors and assistant directors to attend. He gave them the appointment, and they couldn't refuse. Kudos to him, he turned up too, and thoroughly enjoyed the havoc I'd created.

I even had people sending text messages regarding unfulfilled orders, banks wanting surety by the 4th day (hour) and people acting as irate customers etc.

And yes, with the chairmans' blessing, I really did lock the door. If I hadn't they would have given up after an hour, and just gone back to the office. This way, they had to sort out the problems. It gave them a very sharp realisation of what could/can happen, and the impact.

The "lessons learned" document I did as follow up cracked it - I expanded what they'd done to cover a months' shut down. At the (then) current state, it would have bust the company; now, it won't.

Lets just say I gave them a pair of glasses to eradicate the shortsightedness.

They STILL have an annual table top exercise; over the past few years these have included fire, flood, lack of power, lack of water etc. even lack of staff due to an illness epidemic. (That promoted a skills overlap, which they still try to apply)

The only difference now is: no-one locks the door as they no longer run away!

Hope this helps. It takes a helluva lot of planning, and you'll need some backing, but do it as a project in its' own right, and you'll get there.

GG

by jalee1011

seriously great idea.
Ive been trying to get this done. I have managed to do some server consolidation (I was point blanked turned down on purchase of a redundant server for now) and I managed to free up a server to back up my exchange. I also had a backup server that was purchased and was sitting in the corner by my ex-boss that will backup our backoffice system.

I have the two servers, really thats about 80% of our infrastrucure right now. I have a co-esxisting project in bringing up my 100+ remote locations to WAN and eventually will have to broaden the way I handle that data but for now, my focus is on Hq DR.

I wonder if I can lock my CEO, Director of HR and Director of Operations in a room? I might enjoy their discomfort haha.

by ulti

Honestly, IT is one of those parts of a company that are looked at during budget time and only to see where they can cut more from it...And no CEO wants to spend money on something that may never happen like a disaster.

I have helped quite a few companies with Disaster Recovery or the new term of Business Continuity Planning, which seems to work better for ill-informed CEOs.

To me a BCP or DR plan is not just about having proper backups. But if that is where you want to start with your company, it is a good place.

If it is true that your company has $60 mil in annual revenue, then you really need to just write up a detailed plan about how much revenue would be lost during even a partial outage.

So take some time and create a detailed project plan that describes the lose of the server that holds all of the companies accounting data due to a hardware failure. This is coupled with the easy and frequent problem of a bad TAPE that does not allow you to recover the lost data. Now you have a bit of ammo to show that without the proper backup strategy that there is always the possibility of data lose, even in the most controlled environment....which would result in loses of $$$$.

So now you have a figure of how much it will cost to prevent or at least lower the risk of a situation like this, you take this and put it against the lose of all the accounting data. Then you show in your project plan on how much this BCP(DR plan) will SAVE the company. Now you have spun this project from spending money into how much money th company can save!!! Which is what all CEOs really want to see....

This is one of many ways to portray a BCP to a company....

In the end...if they do not want to spend the money on disaster recovery...not much you can do but try and work little changes in where you can...

Good luck, since it is a BIG job to implement a entire BCP for a company and even harder if it is just the IT group that is implementing their own without the other groups working with you.

by jalee1011

I agree with the last part of the small steps at a time.

The $60mil was best guess but it is up there... we operate and franchise 96
c-stores in 3 states.

As far as talk about money, If I cant do what GG in her above post said, and lock my executives in a room, should I analyze the cost that will be incurred (to a best written estimate?) and tell them how much they stand to lose in that 1 event?

Or should I relay my information by informing them the ROI of a BCP plan would be realized within the first recovery x10?

thanks for everyones help!

by hal 9000

For you to work with and see what works best for you.

However I did notice one glaring problem with the Backup's you are not attempting to restore them to different Hardware which you should as many times these can not be performed and the data becomes lost.

The ones that I particularly like is the idea of Work Shopping yearly with a complete disaster floods are always a good one to start off with BTW if you have never done this previously.

But at the same time [b]I Agree Completely[/b] with the Post that says to instead of showing [b]Costs[/b] show [b]Potential Savings[/b] even if you don't get everything that you initially want you'll get much more than if you just stick to the costs of the Implementation as you are showing a potential saving and not a perceived [b]Unnecessary Expense.[/b]

Having been placed in a position of needing to recover Data after a Disaster it takes much longer to explain just why something has been lost than getting a remote Backup and new hardware. Incidentally you could just float the idea of how much it really costs to Recover a Dead HDD Urgently when it fails and takes all the current Data with it. Then point out the delays involved when a Large Scale Incident occurs so you can push the idea that while you are paying [b]Priority Prices[/b] you will be way down the list and instead of taking days it will take weeks or maybe over a month. Working on either Floods or Earthquakes is always a great starting point as both can & do happen with alarming regularity and without warning.

Col

This discussion is closed to new replies.