Community archive

Ask something
Discussion Locked

Solve this disaster scenario

0 helpful
20 replies

by editor's response

Closed to new repliesThis discussion is closed. It stays readable, but it takes no new replies.

Take a look at the scenario below and offer your best solution. The most innovative answers will be noted in the Disaster Recovery e-newsletter on May 6, 2003, along with Mike Talon's advice on this particular topic.

A client recently decided to protect a Microsoft Exchange 2000 system by using a real-time, software-based replication system. He procured a second, identical server with the same amount of storage space, and even added 50 percent more disk to accommodate growth.

Finally, to keep the system humming, he installed disk management software to automatically defrag and maintain the assorted volumes. HA systems were set up at the DR site and tested to ensure that the backup systems would take over within fifteen minutes if theproduction systems fail.

What can go wrong with this solution? Can you spot the major flaws?

If you haven't subscribed to our free DR e-newsletter, be sure to do so today!

https://www.techrepublic.com/techmails.jhtml?repID=r001

20 replies

by lordinfidel

Running disk management software to defrag drives that exchange is on is all sorts of bad.

Exchange has it's own built-in utilities for cleaning/releasing unused space.

In short, defragging an exchange based system, can and will lead to data corruption. Don't do it.

by clearsmashdrop

I second Lord Infidel's assertion you should not run a defrag on Exchange. I learned the hard way that is not a smart thing to do. I ended up corrupting the database. Ooops.

by old and tired..

1.
if it's truly mission critical, 15 mins is a lifetime...we should be thinking REDUNDANT systems which kick in right away!
2.
defrag/maintain is EXCHANGE business today...let it do it's job ... external software can and will cause untold headaches, if not failure(s)
3.
if there are two systems, don't see the point in the "added 50% more disk for growth" in the second system...instead of all this complexity..stick with MIRRORING

by miketalonnyc

Exchange alone can take 15 minutes to come up after a failover, so that's a tight HA solution if the client can promise 15 minutes. Unfortunately there are no SAFE instant failover tools for most Windows-based applications.

The 50% overhead isjust a good idea in many cases, just in case.

Part 2 is on the right track, but don't stop there, the solution requires a bit more thought along those lines.

Mike Talon

by Member 36761510

truncated message continued.....

Taking into account what we know, ensuring that there are no filesytem antivurus scanners (pure exchange mailbox ones are okay) because they leave hooks in the transaction logs, which give you a partially attacheddatabase on the target side. Async replication, which all software replication is (that I have ever used), presents a real probability in data loss if there is a source machine failure. The transactions can be confirmed at the target end and possible database corruption can occur. The busier the system, the more likely there will be data loss.

...The other points about defragging exchange are quite correct - don't do it! Also the 15-minute failover can be reduced if you have auto failure detection in the replication software. This as well can cause problems with split-brain syndrome. The target site assuming a failure has occurred (incorrectly due to a network glitch) and swapping active machine names, where the source machine is actually still okay, when the network comes back - you have 2 machines with the same id and hey presto - lots of problems!!

by Member 36761510

Ok let me explain the title....
Nothing is WRONG with trying to replicate Exchange 2000, but you have to be damn sure you follow the following steps. I am a Consultant working for a Storage Reseller and solutions provider. I specialise in DR and replication as well as backup and SAN etc..

As Mike quite rightly said there are massive costs involved with Hardware replication, you need a 100MB link to keep the in order write sequence at the target end confirming its I/O write before the Source end can do some more work.. That said something like Legato Co-Standby or their new AAM product works magnificently for this. You also need to prepare the disks at the source and target end, that may require (usually) a re-install of the application in question.. lots of unwanted downtime ahead...

To combat this Software replication for about 5 grand per pair is considerably cheaper to install and configure, requires no reinstallation of application and can be done in a few days (with testing of course!!!). Software replication can be throttled and scheduled to only update the target periodically.

Again Mike was correct in saying that async or near sync replication can cause corruption at the target end if the source end fails unexpectedly (usually a failure, not a clean shutdown).

by miketalonnyc

If memory serves me well (sorry, just finished watching the Tivo of King of Iron Chefs), I mentioned that data corruption could occur in improperly confiugred Asyc systems. Most of these solutions offer write order preservation and other protectionmethodologies to prevent corruption even in unexpected shutdowns once they are properly configured.

Don't want the vendors looking to kill me =)

Mike Talon

by miketalonnyc

Modern software-based replication solutions can protect data even during source failure, real-time asychronous data-transfer solutions have come a long way in just the last 18 months or so.

However, you are still just about nailing the major issue addresed in the solution explanation I'll present next week. Keep searching!

Mike Talon

by hal 9000

This is a real one so How about running with this.

A server that is less than 2 weeks old fails and all the computers in the business also fail. When the computer is returned to the maker the customer is told that it has suffered a lighting strike and is not covered by the manfactures warantie and that it should be covered by their insurance company. Upon an Insurance Claim being loged the insurance company requires an independant assesment of the computer and sends it off to their nominatedrepairer for a Inspection Report.

The company who recieves the computer for the Insurance report first opens the case to find a mess of chared circutiry and carbon lying on the bottom of the case all consistent with a severe power surge or lighting strike. But out of courisiorty puts a power supply tester on the main laed and applies power not expecting anything to happen and the tester imediately blows up. Upon closer inspection mains voltage is found coming out of the 12 volt and 5 volt supply as well as the 3 volt lines. It is imeaditly apparent that mains voltage has caused all the damage and upon inspection the repaired finds an unbranded 250 Watt power supply trying to drive a Dual Xeon Process M'Board with 4 IDE devices and 8 SCSI devices connected to the built in SCSI controller on the M'Board.

The repairer them proceeds to the business to assess the rest of the damage and finds everything has suffered the same fate from the humble hubs to every computer on the network all 250 machines.

Now here's the 2 questions

1 What would you do go along with the lie knowing if found out you would be charged with fraud? Report the truth and then expect to spend a long time in court as a wittness for the business to recover costs both real and punitive?

2 How would you go about fixing this situation if you told the truth?

by miketalonnyc

Did this happen to your company? Under the circumstances I'd suggest the first thing you'd want to do is contact a qualified attourney.

Meanwhile, if you'd like this to be a case study for a future scenario in this column, let's talk!

Mike Talon
MikeTalonNYC@yahoo.com

by hal 9000

I got the job from the insurance company and just did a Report stating what I found and a projected repair bill. I left los of earning out of the report as I'm not qualified to make any estimate on this one.

I then left it to the insurance company to deal with as it's really none of my business what hapens next but I don't expect the insurance company to pay out and I do expect to be called as a witness for a court hearing sometime in the future.

As far as contacting a Lawer goes I'm not all that interested as I'm not directly involved I'm only the nominater repairer from the Insurance company and the facts are incontravertiable. I was only asked to prepare a report for the Insurance company but I did have a problem aporting blame for this one as I was unsure exactly who was to blame the company who fitted the ps or the maker of the ps so I split the blame between both as it was vastly underrated for the required job but equally it should not have allowed mains voltage into the case either.

Atherwise I didn't know how to proceed.

If you want to use this seranio your welcome as it does show exactly what can happen when a company who specalises in Home computers attempts to build a High Performance Business Computer. And I'd like to hear some ones else input as well.

by yhoosucksrocks

The type of damage described can all be attributed to lightning damage, I've seen stranger things. Although the systems are under built lightning is still the cause of the damage. The insurance should pay and you should seek out a better vendor for the replacement equipment.

by hal 9000

And since it happened at a time when there was no rain/thunder storms ect I would have to discount this as the cause.
Also if it was a geninue lighting strike the UPS connected to the server should have been destroyed as well. Not to mention all of the other electrical applicences in the building like Fax machines, airconditioning, fridges and the telephones all of these worked without any problems it was only the computer system that was destroyed. If it was a lighting strike it was very selective and it is beyond my ability to believe that this could be the case.
However I do have to agree that the maker of the server in question should be taken out and shot and under no circumstances ever attempt to build a computer of this type of configeration ever again. What I do find interesting is that some one who builds home computers imeaditelly thinkds that they are capable of building a high end business computer just because they have the ability to build game machines. Today I attended a business to have a look see for a potential job that at first only required the fitting of a DVD drive to a new computer as it had no CD/DVD installed and only the HDD and a floppy. The company who supplied it also sold them a second TTF monitor for a second display so that the customer could see what ever as I don't know exactly what they where trying to do but when they came and installed the computer they set it up to the network, set everything else up and declined to fit the second monitor.

by pipe guy

Maybe I'm showing my age when I say that a 250 watt powersupply used to be enough to power dual processors and a case full of scsi drives and decent server components. I recently changed a powersupply that had been in service for 7 years which was exactly 240 watts and did everything required to spec. I must add that the new powersupply even though it was a 450 watt ps, was about 2/3rds the weight of the old power supply. So perhaps when a customer paid 6,000 for the first Pentium 2 server in the mid 90s they got what they paid for. I suspect if those new 250 watt powersupplies had been tested under load they couldn't produce more than 200 watts and maintain the voltage required for the hardware. Obviously with today's hardware no consultant worth his salt would spec anything less than 350 or 400 watts for any P4 type processor. I didn't think that they were even manufactured anymore.

by pipe guy

Maybe I'm showing my age when I say that a 250 watt powersupply used to be enough to power dual processors and a case full of scsi drives and decent server components. I recently changed a powersupply that had been in service for 7 years which was exactly 240 watts and did everything required to spec. I must add that the new powersupply even though it was a 450 watt ps, was about 2/3rds the weight of the old power supply. So perhaps when a customer paid 6,000 for the first Pentium 2 server in the mid 90s they got what they paid for. I suspect if those new 250 watt powersupplies had been tested under load they couldn't produce more than 200 watts and maintain the voltage required for the hardware. Obviously with today's hardware no consultant worth his salt would spec anything less than 350 or 400 watts for any P4 type processor. I didn't think that they were even manufactured anymore.

by markhoenig

It seems to be implied. If so, many disasters (ie fire) would take out both servers. If not, then we also have to look at the connectivity/bandwidth.

by plaporte

It seems to me that one element of the discussion that's missing is to ask, "What event(s) is the person seeking to protect against?" The on target technical answers need to be matched with the person's goals in initiating additional level of protection or coverage. For example, a person requiring near-instant failover, but unconcerned with facility, network or region disasters would opt for local clustering. But this would be the wrong, or only a partial solution, if the person was looking to comprehensively protect email from all threats, internal and external.
Answering the "What's the goal?" question frames up the discussion to better evaluate trade-offs.

by hal 9000

But with the computer there where a bunch of CD's that included the OEM Windows XP disk a recovery disk the monitor disk {it is an ebook} and a few others but I don't know exactly how you would ever be able to actually use them in the supplied computer but they would make good platters for tea cups and the like. The computer only came with XP Pro installed and nothing else so to all intents and purposes it wasn't much use to the company.
The network was also set up with a broadband modem connected directly to the hub and then every computer ran off this hub with no firewall or any virus protection and there was no means provided for performing data backup's unless of course you where prepared to back everything back to floppies. There wasnetwork cabeling hanging out of the walls unsecured and the whole thing was a general mess from a business point of view.
But I suspose you could always log onto another computers CD drive and run what ever from there but that would not be much help if the thing fell over and needed to be reloaded or even have the Windows XP disk inserted to do a repair. AS well there was no printer available across the entire network but in all fairness they had only just installed the computer 2 weeks ago and where going to come back and finish the rest of the job sometime in the future . I suspose that this is acceptiable for home use but I don't know of many business that could be very productive this way.
The only way that they could print anything was to e-mail it to the station with the only printer in the place and then open the e-mail and print it from that workstation. To my way of thinking it sought of defeats the whole idea of having a network in the first place.

This discussion is closed to new replies.