General discussion

  • Creator
    Topic
  • #2307066

    Solve this disaster scenario

    Locked

    by editor’s response ·

    Take a look at the scenario below and offer your best solution. The most innovative answers will be noted in the Disaster Recovery e-newsletter on May 6, 2003, along with Mike Talon’s advice on this particular topic.

    A client recently decided to protect a Microsoft Exchange 2000 system by using a real-time, software-based replication system. He procured a second, identical server with the same amount of storage space, and even added 50 percent more disk to accommodate growth.

    Finally, to keep the system humming, he installed disk management software to automatically defrag and maintain the assorted volumes. HA systems were set up at the DR site and tested to ensure that the backup systems would take over within fifteen minutes if theproduction systems fail.

    What can go wrong with this solution? Can you spot the major flaws?

    If you haven’t subscribed to our free DR e-newsletter, be sure to do so today!

    http://www.techrepublic.com/techmails.jhtml?repID=r001

All Comments

  • Author
    Replies
    • #3520874

      Disk Mgmt Software

      by lordinfidel ·

      In reply to Solve this disaster scenario

      Running disk management software to defrag drives that exchange is on is all sorts of bad.

      Exchange has it’s own built-in utilities for cleaning/releasing unused space.

      In short, defragging an exchange based system, can and will lead to data corruption. Don’t do it.

      • #3521473

        Defragging Exchange

        by clearsmashdrop ·

        In reply to Disk Mgmt Software

        I second Lord Infidel’s assertion you should not run a defrag on Exchange. I learned the hard way that is not a smart thing to do. I ended up corrupting the database. Ooops.

    • #3521540

      15 mins?

      by old and tired.. ·

      In reply to Solve this disaster scenario

      1.
      if it’s truly mission critical, 15 mins is a lifetime…we should be thinking REDUNDANT systems which kick in right away!
      2.
      defrag/maintain is EXCHANGE business today…let it do it’s job … external software can and will cause untold headaches, if not failure(s)
      3.
      if there are two systems, don’t see the point in the “added 50% more disk for growth” in the second system…instead of all this complexity..stick with MIRRORING

      • #3358143

        We’re talking Windows here

        by miketalonnyc ·

        In reply to 15 mins?

        Exchange alone can take 15 minutes to come up after a failover, so that’s a tight HA solution if the client can promise 15 minutes. Unfortunately there are no SAFE instant failover tools for most Windows-based applications.

        The 50% overhead isjust a good idea in many cases, just in case.

        Part 2 is on the right track, but don’t stop there, the solution requires a bit more thought along those lines.

        Mike Talon

    • #3521233

      Everything and nothing continued…

      by pritesh_mehta ·

      In reply to Solve this disaster scenario

      truncated message continued…..

      Taking into account what we know, ensuring that there are no filesytem antivurus scanners (pure exchange mailbox ones are okay) because they leave hooks in the transaction logs, which give you a partially attacheddatabase on the target side. Async replication, which all software replication is (that I have ever used), presents a real probability in data loss if there is a source machine failure. The transactions can be confirmed at the target end and possible database corruption can occur. The busier the system, the more likely there will be data loss.

      …The other points about defragging exchange are quite correct – don’t do it! Also the 15-minute failover can be reduced if you have auto failure detection in the replication software. This as well can cause problems with split-brain syndrome. The target site assuming a failure has occurred (incorrectly due to a network glitch) and swapping active machine names, where the source machine is actually still okay, when the network comes back – you have 2 machines with the same id and hey presto – lots of problems!!

      • #3521203

        Evrything and Nothing

        by pritesh_mehta ·

        In reply to Everything and nothing continued…

        Ok let me explain the title….
        Nothing is WRONG with trying to replicate Exchange 2000, but you have to be damn sure you follow the following steps. I am a Consultant working for a Storage Reseller and solutions provider. I specialise in DR and replication as well as backup and SAN etc..

        As Mike quite rightly said there are massive costs involved with Hardware replication, you need a 100MB link to keep the in order write sequence at the target end confirming its I/O write before the Source end can do some more work.. That said something like Legato Co-Standby or their new AAM product works magnificently for this. You also need to prepare the disks at the source and target end, that may require (usually) a re-install of the application in question.. lots of unwanted downtime ahead…

        To combat this Software replication for about 5 grand per pair is considerably cheaper to install and configure, requires no reinstallation of application and can be done in a few days (with testing of course!!!). Software replication can be throttled and scheduled to only update the target periodically.

        Again Mike was correct in saying that async or near sync replication can cause corruption at the target end if the source end fails unexpectedly (usually a failure, not a clean shutdown).

        • #3358134

          one note

          by miketalonnyc ·

          In reply to Evrything and Nothing

          If memory serves me well (sorry, just finished watching the Tivo of King of Iron Chefs), I mentioned that data corruption could occur in improperly confiugred Asyc systems. Most of these solutions offer write order preservation and other protectionmethodologies to prevent corruption even in unexpected shutdowns once they are properly configured.

          Don’t want the vendors looking to kill me =)

          Mike Talon

      • #3358140

        Keep going

        by miketalonnyc ·

        In reply to Everything and nothing continued…

        Modern software-based replication solutions can protect data even during source failure, real-time asychronous data-transfer solutions have come a long way in just the last 18 months or so.

        However, you are still just about nailing the major issue addresed in the solution explanation I’ll present next week. Keep searching!

        Mike Talon

    • #3358118

      Here’s a better Disater Senario

      by hal 9000 ·

      In reply to Solve this disaster scenario

      This is a real one so How about running with this.

      A server that is less than 2 weeks old fails and all the computers in the business also fail. When the computer is returned to the maker the customer is told that it has suffered a lighting strike and is not covered by the manfactures warantie and that it should be covered by their insurance company. Upon an Insurance Claim being loged the insurance company requires an independant assesment of the computer and sends it off to their nominatedrepairer for a Inspection Report.

      The company who recieves the computer for the Insurance report first opens the case to find a mess of chared circutiry and carbon lying on the bottom of the case all consistent with a severe power surge or lighting strike. But out of courisiorty puts a power supply tester on the main laed and applies power not expecting anything to happen and the tester imediately blows up. Upon closer inspection mains voltage is found coming out of the 12 volt and 5 volt supply as well as the 3 volt lines. It is imeaditly apparent that mains voltage has caused all the damage and upon inspection the repaired finds an unbranded 250 Watt power supply trying to drive a Dual Xeon Process M’Board with 4 IDE devices and 8 SCSI devices connected to the built in SCSI controller on the M’Board.

      The repairer them proceeds to the business to assess the rest of the damage and finds everything has suffered the same fate from the humble hubs to every computer on the network all 250 machines.

      Now here’s the 2 questions

      1 What would you do go along with the lie knowing if found out you would be charged with fraud? Report the truth and then expect to spend a long time in court as a wittness for the business to recover costs both real and punitive?

      2 How would you go about fixing this situation if you told the truth?

      • #3358064

        Is this real?

        by miketalonnyc ·

        In reply to Here’s a better Disater Senario

        Did this happen to your company? Under the circumstances I’d suggest the first thing you’d want to do is contact a qualified attourney.

        Meanwhile, if you’d like this to be a case study for a future scenario in this column, let’s talk!

        Mike Talon
        MikeTalonNYC@yahoo.com

        • #3357897

          Yes unfortantly it is

          by hal 9000 ·

          In reply to Is this real?

          I got the job from the insurance company and just did a Report stating what I found and a projected repair bill. I left los of earning out of the report as I’m not qualified to make any estimate on this one.

          I then left it to the insurance company to deal with as it’s really none of my business what hapens next but I don’t expect the insurance company to pay out and I do expect to be called as a witness for a court hearing sometime in the future.

          As far as contacting a Lawer goes I’m not all that interested as I’m not directly involved I’m only the nominater repairer from the Insurance company and the facts are incontravertiable. I was only asked to prepare a report for the Insurance company but I did have a problem aporting blame for this one as I was unsure exactly who was to blame the company who fitted the ps or the maker of the ps so I split the blame between both as it was vastly underrated for the required job but equally it should not have allowed mains voltage into the case either.

          Atherwise I didn’t know how to proceed.

          If you want to use this seranio your welcome as it does show exactly what can happen when a company who specalises in Home computers attempts to build a High Performance Business Computer. And I’d like to hear some ones else input as well.

        • #3357382

          Link for reference

          by cactus pete ·

          In reply to Is this real?

          Mike-

          I’ll save some typing, perhaps, by everyone and post the link to colinluck’s previous entry on this matter from an earlier thread having to do with ethics:

          http://www.techrepublic.com/forumdiscuss/thread_detail.jhtml?thread_id=121068&message_id=731388

          Please remember to remove the inevitable space in the url…

      • #3357313

        Sounds like lightning to me

        by yhoosucksrocks ·

        In reply to Here’s a better Disater Senario

        The type of damage described can all be attributed to lightning damage, I’ve seen stranger things. Although the systems are under built lightning is still the cause of the damage. The insurance should pay and you should seek out a better vendor for the replacement equipment.

        • #3359017

          Sorry but I did the report

          by hal 9000 ·

          In reply to Sounds like lightning to me

          And since it happened at a time when there was no rain/thunder storms ect I would have to discount this as the cause.
          Also if it was a geninue lighting strike the UPS connected to the server should have been destroyed as well. Not to mention all of the other electrical applicences in the building like Fax machines, airconditioning, fridges and the telephones all of these worked without any problems it was only the computer system that was destroyed. If it was a lighting strike it was very selective and it is beyond my ability to believe that this could be the case.
          However I do have to agree that the maker of the server in question should be taken out and shot and under no circumstances ever attempt to build a computer of this type of configeration ever again. What I do find interesting is that some one who builds home computers imeaditelly thinkds that they are capable of building a high end business computer just because they have the ability to build game machines. Today I attended a business to have a look see for a potential job that at first only required the fitting of a DVD drive to a new computer as it had no CD/DVD installed and only the HDD and a floppy. The company who supplied it also sold them a second TTF monitor for a second display so that the customer could see what ever as I don’t know exactly what they where trying to do but when they came and installed the computer they set it up to the network, set everything else up and declined to fit the second monitor.

        • #3359015

          Part 2

          by hal 9000 ·

          In reply to Sorry but I did the report

          But with the computer there where a bunch of CD’s that included the OEM Windows XP disk a recovery disk the monitor disk {it is an ebook} and a few others but I don’t know exactly how you would ever be able to actually use them in the supplied computer but they would make good platters for tea cups and the like. The computer only came with XP Pro installed and nothing else so to all intents and purposes it wasn’t much use to the company.
          The network was also set up with a broadband modem connected directly to the hub and then every computer ran off this hub with no firewall or any virus protection and there was no means provided for performing data backup’s unless of course you where prepared to back everything back to floppies. There wasnetwork cabeling hanging out of the walls unsecured and the whole thing was a general mess from a business point of view.
          But I suspose you could always log onto another computers CD drive and run what ever from there but that would not be much help if the thing fell over and needed to be reloaded or even have the Windows XP disk inserted to do a repair. AS well there was no printer available across the entire network but in all fairness they had only just installed the computer 2 weeks ago and where going to come back and finish the rest of the job sometime in the future . I suspose that this is acceptiable for home use but I don’t know of many business that could be very productive this way.
          The only way that they could print anything was to e-mail it to the station with the only printer in the place and then open the e-mail and print it from that workstation. To my way of thinking it sought of defeats the whole idea of having a network in the first place.

      • #3356533

        250 watts used to be enough

        by pipe guy ·

        In reply to Here’s a better Disater Senario

        Maybe I’m showing my age when I say that a 250 watt powersupply used to be enough to power dual processors and a case full of scsi drives and decent server components. I recently changed a powersupply that had been in service for 7 years which was exactly 240 watts and did everything required to spec. I must add that the new powersupply even though it was a 450 watt ps, was about 2/3rds the weight of the old power supply. So perhaps when a customer paid 6,000 for the first Pentium 2 server in the mid 90s they got what they paid for. I suspect if those new 250 watt powersupplies had been tested under load they couldn’t produce more than 200 watts and maintain the voltage required for the hardware. Obviously with today’s hardware no consultant worth his salt would spec anything less than 350 or 400 watts for any P4 type processor. I didn’t think that they were even manufactured anymore.

      • #3356532

        250 watts used to be enough

        by pipe guy ·

        In reply to Here’s a better Disater Senario

        Maybe I’m showing my age when I say that a 250 watt powersupply used to be enough to power dual processors and a case full of scsi drives and decent server components. I recently changed a powersupply that had been in service for 7 years which was exactly 240 watts and did everything required to spec. I must add that the new powersupply even though it was a 450 watt ps, was about 2/3rds the weight of the old power supply. So perhaps when a customer paid 6,000 for the first Pentium 2 server in the mid 90s they got what they paid for. I suspect if those new 250 watt powersupplies had been tested under load they couldn’t produce more than 200 watts and maintain the voltage required for the hardware. Obviously with today’s hardware no consultant worth his salt would spec anything less than 350 or 400 watts for any P4 type processor. I didn’t think that they were even manufactured anymore.

    • #3357988

      Are both servers at the same location?

      by markhoenig ·

      In reply to Solve this disaster scenario

      It seems to be implied. If so, many disasters (ie fire) would take out both servers. If not, then we also have to look at the connectivity/bandwidth.

    • #3357383

      hello

      by ashish_nehra1 ·

      In reply to Solve this disaster scenario

      this is for test

    • #3359022

      What’ the goal?

      by plaporte ·

      In reply to Solve this disaster scenario

      It seems to me that one element of the discussion that’s missing is to ask, “What event(s) is the person seeking to protect against?” The on target technical answers need to be matched with the person’s goals in initiating additional level of protection or coverage. For example, a person requiring near-instant failover, but unconcerned with facility, network or region disasters would opt for local clustering. But this would be the wrong, or only a partial solution, if the person was looking to comprehensively protect email from all threats, internal and external.
      Answering the “What’s the goal?” question frames up the discussion to better evaluate trade-offs.

Viewing 6 reply threads