26 replies
by gary dunlap
I too have been given this task. If I lay my hands on something I will forward you a copy.
Gary Dunlap
Clay County Sheriff's Office
gdunlap@claysheriff.com
by al macintyre
Most major computer conferences include several seminars on this kind of subject with wonderful hand-outs of check lists of considerations. Sometimes you can locate the speakers & get copy without going to the seminar.
www.common.org probably does the best ones, but they are expensive if the only thing you want from the conference is this topic.
Most major vendors like IBM have classes in a topic called System Administration, which includes such stuff as the types of backups we should bemaking, how to recover from mishaps, overall Availability & what we can do to increase this, ie. reduce risk that we will have an internal disaster, as opposed to the flood tornado earthquake variety.
In the wake of latter hitting some area, there are often articles regarding how those kinds of places coped ... like now your computer room is under water, what can you do to make it practical to salvage what was water logged (for some platforms this is doable, for others forget it).
http://www.ifccfbi.gov might be one place to check on overall responsibilities, expectations, contacts in law enforcement
http://www.itpolicy.gsa.gov/ is your on ramp to government white papers for a spectrum of issues.
Perhaps some of these posts will give you a good starting point.
by kwangern
by al macintyre
How often do you back up everything? How do you know everything is on your backups? Do you rotate a set of backups off site so that in the event of a physical disaster you have access to all your software & records? Is the hardware that your backups go to such that they can be replaced easily?
Example - friends of mine involved in IT with a major doctor office. Doctor office got hit over weekend with theft of hardware. No problem they thought, we have off site copy all our stuff & a list of our equipment & software - insurance will pay for us to get replacements, then we will just restore to the replacement stuff our off-site backups. No, they found that their backup technology was now old, they could not get any way to read theirbackup tapes.
Assuming you do like us, with backup copies of key equipment, so that if it breaks down, we plug in the backup copy then send the broken copy in for repair, except we are not storing this off-site, which is a risk ... is your off-site list of hardware & software up-to-date?
by al macintyre
Backup Media stored RIGHT BESIDE the computer that is being backed up means that in the event of a physical disaster - fire tornado, etc. you lose both the computer & the backup.
There are several other issues. Suppose there is turn-over & the former employee walks off with some of the stuff on the computer. Do you have an inventory of what belongs to the employee & what belongs to the enterprise?
Who knows where the backups are located ... I think the normal user of the device, the people involved in doing the backups, the head of the department.
First issue is just to be backing up everything regularly. Do you have new equipment that came with proper provision for backups? Once upon a time we had a guy going around to all equipment plugging a tape drive into printer port to do a few backups a nite - very tedious.
Now we have PCs on a network with backups for the connected PCs via the network but is everything really being backed up or do users have stuff on client that is not.
Who audits that everything is backed up & how much of a hassle is this - how often is that done?
Then there is the issue of where & how to store the backup media for a variety of reasons. You need convenient access, you need distance in case of disaster, you need to know how to store it so the tape is usable.
by al macintyre
You may need to know WHEN backup like TIME because if you restore to close of business 2 days ago, who knows what they did since that now they will have to do again?
Do you refrain from deleting audit trails until after the next backup?
Do youmaintain any control totals of daily activity so if you have to do the day's work a second time, in the absense of the paperwork involved, how will you know you got it right?
The techniques will be different for different applications ... you paid those bills - you need to flag them as paid but not pay the vendors a second time ... these others entered ... may need to take a physical inventory rather than try to replicate all inventory transactions.
by project 3000
I have found some very good articles and sample plans at the following website
http://www.labmice.net/disaster.htm
Mark Evans
by al macintyre
http://www.labmice.net/disaster.htm is a GREAT collection of resources ... Thanks Mark.
It has sample plans, links to FEMA & other groups, specialists with "drill down" yellow pages by category, relevant publications, free subscriptions, tutorials, outlines, terminology, relevant technologies, different types of disasters bring different types of problems - all described here, lots of articles.
You want to be sure to book mark this site then go browsing through its resources. However, the fact that it is aimed at a Windows 2K audience & Novell LAN & other "popular" platform mixtures implies that folks on other platforms may need some resources in addition to what this site offers.
by al macintyre
by al macintyre
Protect your assets. Protect access to your assets.
Environment: Equipment at multiple sites; backup of connected & unconnected PCs & host on-line & off-line data in a mixture of locations; Off-site backup not just of host, but some PC applications (e.g. Payroll & Auto Cad). People take portable PCs around. Suppose one is stolen - do you know what confidential stuff was on there like passwords? Will that be reported promptly to the people who need to change their passwords.
Equipmentcan be borrowed legitimately within the enterprise but some people not know it was borrowed. One enterprise puts bar codes on all equipment that is portable then someone walks through office regularly taking inventory - compare to last inventory - compare to official inventory - whats missing - which was last department to have it in inventory that last had it listed.
Air Conditioning: separate system; protected intake; fire dampers.
Power: Uninterruptable power supply distributed beyond computer room; Surge protection; sabotage protection via controlled access; overload safety considerations.
Detection & Alarm systems: Early warning; Alert someone if disaster during hours that no-one is in the building. If a water sprinkler dowses a computer peripheral with water, will that cause an electrical short?
Some systems douse computer with halon or other oxygen remover ... how are humans protected?
Training: Fire Extinguishers; what to do in various types of emergencies;
by al macintyre
Exposures: Fire; Water Damage; Tornado; Flood; Utility failure; Plane crash into building; Vandalism; Information loss-alteration-destruction - deliberate or accidental; key information in mind of MIS personnel who can suddenly depart from the company through death-accident--turnover (e.g. security, day-to-day outage recovery).
I give my managers a briefing on seucrity - they put passwords to the security administrator in a sealed envelope locked in a safe so that if the security adminstrator accident happens, some other professional can get into the system.
I had a friend who worked at one of those places in which the doors are computer controlled & they had a major diskaster. The building was on fire, the computers were down, thedoors would not work, they had to use fire axes to get their way out of the building. There needs to be a special kind of consultant to make sure your people can find their way out of the building when the doors are locked & the lights are out.
Once upon a time janitor was in building & I did not know it & I lccked it up & set the alarm system ... incidentally when employees who know the alarm system codes leave the company, do you change their codes?
by al macintyre
Buildings: Fire resistant construction; fire suppression systems (automatic & hand-held); forms storage away from equipment; limit exposure to hazards (interference by power cables, industrial machinery, burst water pipe vs. shut-off valves) with suitable covers; availability of municipal services in reasonable time period; access control; fire walls; backup media far enough away from on-line storage so don't lose both in one physical disaster; restricted access without locking people inside. give some thought to measures not taken by victims of national disasters (earthquake in California, Chicago infrastructure, Mississippi river flooding) that had individual businesses taken better precautions they would have been more likely to survive.
Storage containers: fire-proof and heat-proof cabinets for diskettes and engineering drawings - if a building burns to the ground, what will be the condition of the contents of supposedly fire-proof file cabinets? If we have such furniture, what else belongs in them?
Maks sure you know what is heat proof (diskettes don't melt) and what is fire proof (papers not singed).
Does the enterprise have an emergency evaculation plan with exit signs that are clearly functional when the building power is out? Is this plan ever tested?
At home I have flash lights plugged into wall sockets such that when the utility power goes out, the flash light turns on, I can navigate, I can grab one ... does your business have that kind of thing for inside areas not close to windows?
Does community have some emergency sirens radio when trouble & can your workers access whatever the story is about what is going on.
Do you have posted near phones & in bathroom etc. guidance on what to do if someon poisoned? Does your receptionist know what to do if someone phones in a bomb threat?
by al macintyre
* IDENTIFY AND PRIORITIZE VITAL APPLICATIONS
Contingency Planning: top-down management support & decide degree of protection needed; identify
and prioritize vital applications; develop strategies to limit damage caused by a disaster, keep vital jobs of the
business running, restore operation to its prior status; document the plan; test the plan; update the plan.
Vital application = an application processing and producing information critically important to the
organization - that must continue as quickly as possible after a catastrophe.
Vital information = information without which the organization could not operate after a disaster;
Consider scenario of web site continuing to accept customer orders before the company has recovered ... is that being backed up properly so you not lose any orders & are the folks on the web site told that your situation is what?
by al macintyre
Financial loss: records on inventory, work-in-process and requirements; accounting data - bills which
have been paid, receivables, regulatory compliance.
Legal commitments: shipments; contracts; payroll; payables; taxes.
Business disruption: inventory records; calendar appointments meetings; production schedules; ISO &
UL standards to be met; document distribution.
Involve: Users of Information; Supplier or originators of information; management responsible for
applications; providers of service (MIS, PC repair, public utilities); auditor; insurance carriers.
Determine: period of time application can be disrupted without irretrievable damage; loss exposure of
application according to length of disruption; acceptablelevel of downgraded service; acceptable period of time
for downgraded service level.
by al macintyre
Identify vital records requiring back-up: programs-software; files-data diskettes CD Rom; hard copy listings
(e.g. engineering drawings); documentation of operational procedures; forms (e.g. invoices, checks, shop forms); documentation needed to run the business as required by regulators & various certifications.
Identify key personnel for each vital application. How can you reach them when normal channels are disrupted?
Off-site storage of vital records on a continuing basis.
by al macintyre
I am now subscribed to this discussion thread so if there is other activity it will remind me that I never got finished.
I still have to do
* THE EMERGENCY PLAN = BACKUP -> RECOVERY
On the other hand, if no other postings then I will realize I misconstrued the need to share this kind of info.
by s snow
Al, thanks for the notes.
You would probably get to this, but here's my two cents. After all the planning is finished, be sure to actually test your plan. We did a surprise meeting - when everyone showed up, they were told what disaster we had just experienced - we then went through all the steps to be sure our plan worked. We spent about 3-4 hours and learned a great deal. The facilitator needs to push everyone's assumptions. In our scenario, we used an explosion. Just pick something that will get everyone thinking.
by al macintyre
The Red Cross has a calendar filled with a check list of advice like what every first aid kit should be stocked with, other home security advice, various health tips.
In my home, into some wall plugs, I have flash lights whose batteries are regenerated when the power is there & light up when the power is gone.
How often do you check the smoke alarms in your home (required by law to do it once a month at home & our maintenance people supposed to do it when they do quarterly maintenance) but how about at the office ... do you even know where the fire extinquishers are located.
When you travel, do you test the smoke alarms at your hotel & always ask for a room no higher than 7th floor (because fire engine ladders usually go no higher)?
by al macintyre
by al macintyre
Limit damage caused by a disaster: early warning detection; people are the first priority; evacuation;
shelter; containment; suppression; testing.
Keep the business running: user responsibility & management review; identification of vital
applications and discretionary jobs; identification of vital records. Determine: period of time application can be
disrupted; acceptable level of downgraded service-time; cost of recovery of application-time. Set priorities. Set-
up off-site storage of vital records and required documentation; identify key personnel and fallback alternatives.
Resources required: personnel (number, nature, availability); equipment (hardware inventory -
alternative substitutions, software inventory - operating environments to support); logistics (movement
materials & personnel, transportation, schedules, risks); materials (forms, special items, supplies, media -
different kinds of diskettes); communications; vial records off-site. Security audit control = need for increased
controls in outside environment. Backup service alternatives (short & long term): inter-company reciprocal
agreements; mutual-aid agreement; vendor facilities; data servicers; company controlled multiple sites; new
location; manual methods; temporary facilities support (food, sanitation, power supply, transportation,
phones). Testing the backup plan: involve users; by application; third party reviews; modify & document
changes; alternate procedures; annually.
None of this is going to work unless top management has signed off on the resources to make it work.
by al macintyre
The recovery plan:
I, or someone here may need or want to do postings that individually focus on: Backups; Insurance; Local Area Power Protection = LAP; Uninterruptable power supply = UPS;
How do we know we have fully recovered?
I had a minor problem last week ... SURPRISE SOMEONE DELETED SOME OF YOUR SOFTWARE THAT END USERS NEED REGULARLY ... well I restored from backup a copy that preceeded the disappearance & tested that the restore was working ... but normally we only find out thatsomething is missing or broken when someone tries to use it which could be after the backups in rotation have all been used up.
Do we have some kind of inventory of what we should have on the computer system & some way of checking it without human check box each item ... like a backup tape that does not have a backup ... just a list of what we should have ... then some software that generates the list ... so you get new list of what we have now & there is some other software to compare it towhat we have now & list the differences.
A branch of change management software.
I am on AS/400 & recently have been experimenting with *OUTFILE to Query/accessibility. I realize the tools will vary per platform ... I am trying to be technology neutral in describing what needs to be done, then once collectively we think we have everything, there can be focus on HOW DO THIS on particular platforms.
by kgreenizan
This is more than the requirements of your information systems - which are important - but also the communications you will use to keep track of your personnel, your equipment needs and the response teams at the time of the incident. Also you mustconsider the communications needed for the recovery operations once all the damage has been contained.
Communications with the ERC should include updates with or to the local media. A specific individual should be trained in handling the media andbe the sole source for them. In that way you can ensure that proper information is passed and not guesses that could damage your reputation.
by al macintyre
It takes a certain talent to plan the scenario & manage the people playing in it & knowledge of what will go wrong.
I have a friend who works in the military simulations ... there was a scenario in which he tells the judge ... I will throw a handgrenade into that window well of that civilian building just outside the military base. Well the judge does not know what that building houses or its significance to the base & has to find out then communicate to the other judges.
My friend happened to work in computer security & knew that the building housed the ma bell switching for communications & that a very small blast in the concrete window well would direct the explosion into equipment in the basement. So once the judge saw this, he had to tell all the other judges to tell their players that everyone using the cellular & land line civiian communication system, for voice or computer communications - the communications were down for the duration of the simulation game.
No actual damage done, this is a particular kind of simulation game that requires authority for the judges access to places & really well organized test.
I have known real life tests where auditor comes in & literally pulls the plug on computer network & then asks to see how long until we recover ... days of disruption in which end management result is that they are opposed to a particular kind of test.
Different kind of test than the computer stuff, but in our area the medical emergency police fire etc. community does a simulation every few years in which it is declared there has been an earthquake or tornado or something - these roads & bridges are out of commission, we have these people playing dummies to be delivered to hospitals - how fast can you get the job done when key roads are cut?
by pcavalloro
by al macintyre
My Building focus is on actions to reduce risk of a disaster of a preventable accident nature.
Entire building should have regular fire safety checks (extinguishers up to date & exits clearly marked even when the lights go out), but computers have special needs. Building should also be reasonably fire proof without making it more at risk to earthquakes or other disasters.
If high secure areas need access by key cards, provision needs to be made for egress of people inside in the event ofpower failure or other disaster. However, if fire doors can be propped open, or their alarm deactivated by some expert with a screw driver, the exit path needs to be monitored by human security so that assets cannot be wheeled out by someone abusing the human protection arrangements.
Cables under false floors & behind walls ? are they at risk of contact with water leaks? Do computer signal cables have proper distance from power lines?
When we install our UPS, it is of correct strength for current & projected needs, but is this reviewed as we upgrade & add equipment? Our UPS serves the Main Tower & controlling monitor & communications gear ? we left the printer off since it draws a lot of power & we considered it non-essential in an emergency.
Many machines are air cooled ? can the air get there or is it blocked off by adjacent furniture?
Computer forms are stored, not touching equipment where overheating might become a fire hazard, then moved into computer room a day ortwo before opened, to get accustomed to same humidity as where they will be printed on, so as to discourage jams.
by al macintyre
Some communities have lock boxes near large building complexes ? fire & police personnel have keys to the lock boxes. The lock boxes contain blue prints of the buildings & a roster of residents or businesses, so that emergency crews know where to anticipate personnel & what kind of material storage. There is an issue of pass-key to a structure being accessible to emergency crews but secure against normal burglars.
In our community, emergency crews are ?staged? around the city near major highway intersections, so that as they are needed, their travel time is minimized. Then as one ?staged? group goes into service, the rest of the crews have a little dance of placement to continue future community placement optimization based on the total forces out there. There is software to help manage this kind of thing & I would hope every community has something similar.
This discussion is closed to new replies.
Trending IT Employment posts
Sort by Trending