Data Centers

Not so gentle reminder: Business continuity lessons from a northeaster

Bob Eisenhardt compiled some questions for SMBs to ask themselves as they implement business continuity planning. The recent power outages in the northeast bring them into focus.

When will power return to one of my client locations that holds three medical offices? For the past two days this has been a worry, and short of driving 40 minutes and staring into darkness, I have little to do at these locations. When it does return, I may have PLENTY to do, so pending that, I thought I would review some business continuity planning (BCP) questions I have compiled for small business organizations to consider — and relate them to the recent Nor'easter that wrecked the East Coast, but not Halloween!

Question 1: Who can declare a "disaster" or a service interruption? This should be easy - the local business owner can do that, but there must be a chain of command here. If the primary owner is on vacation in China, that makes communication difficult. Two or three people must be able to close the business and implement plans as circumstances dictate. (Real world: text messages do not suffice; they can be ignored. Phone chain works best). Question 2: If your employees cannot report to your location, how long can you afford to wait to re-establish business levels? 24 - 48 - 72 hours? While the medical complex is now closed for two days, medical offices have a standing supply of incoming patients so the first two days will be madness. Pure retail operations, however, do not have a line at the door! They can suffer irreparable loss and may have to close their business. Evaluations must be made of such a circumstance. One week? It can happen. Question 3: Who among your staff knows the most about implementation of emergency procedures? The owner should delegate this to one or two people below them in the hierarchy to co-cover the situation and share responsibilities. These procedures should be WRITTEN DOWN and distributed to staff as part of the employee handbook. Question 4: Do your employees know what to do if they cannot get to work? This begs the question of a co-location. In my medical clients, one is a large optical house and I also support a competitive practice about a mile away. Bluntly, they HATE each other BUT in the event of ONE closing for a significant period of time AND pending PATIENT emergencies, I can put one part of one into the other for a short time. This option requires courage and guts. But for the emergency care patient, it is absolutely VITAL to implement. Question 5: Can you process payroll and financial transactions from your recovery site? As their IT department, my resources can be quickly set up for basic accounting and patient access with a 24 hour window to uptime. It's been tested and verified over the years. I know how their networks are built and how the software runs.

How it turned out this time

It was at this point in writing that I took a break, and at 6:51 Tuesday night power came back on. I was able to RDP into a server. I immediately called and notified my clients by phone and email that power was restored. Then, after a little sleep and early in the morning, I was on the road with coffee to see my client's offices BEFORE staff arrived and mitigate any issues as they arose. There were, thankfully, only a few.

There were no server failures —very thankful there. No workstation failures either, save one critical station that managed only 31,000 odd retina scans; it suffered an OS crash. This was a Windows OS failure and this particular computer, being HIPAA certified, was one I never touched because I was actually forbidden to so. After the usual diagnostics, the vendor was contacted and a replacement system should be arriving tomorrow morning. I then provided the client with the storage path and login information to the server as all 31,000 odd images are server managed. Their technician will need this data ASAP.

There was one printing failure, easily corrected. At the second office, a very inexpensive 5-port switch died and was easily replaced during the day. The third office suffered no problems at all, being just two Windows stations connected via wireless router.

Throughout this, my business continuity planning and familiarity with the office environment ensured that I could manage almost any contingency that arose. DR procedures could be put into place within 24 hours as I also have a plethora of desktop systems available for immediate placement as needed.

A positive upshot of this was that my largest client is now convinced to proceed to a gigabit network, with an estimate presented and to be implemented during the next two weeks.

Summary: Disaster recovery was not an issue in this circumstance and business continuity protocols were firmly enough in place that even the retina image failure is being managed in a timely manner. Oddly enough, the primary vendor of this product is without power as of this writing!

Have you already tested your BCP measures after this early season storm?

Editor's Picks