A disaster recovery plan is a prudent thing to have, but it’s just words on paper until a real disaster comes along to put it to the test. On March 28 at about 6:30 P.M., companies in Fort Worth, TX, had the dubious honor of having their plans severely tested when a tornado hit the city—ripping the glass and aluminum facades off the city’s corporate centers and skyscrapers.
At least two Fort Worth companies discovered that having good plans—and good people to implement those plans—were key elements in keeping their networks up and running while havoc ruled the Texas town.
What worked for them and what didn’t? IT professionals at Cash America International and Union Pacific Resources Group Inc. tell their “disaster survival” stories.
Survival, not recovery
Bill Horne is executive vice president for Information Technology for Cash America, the world’s largest pawn loan company, with almost 500 shops in 20 states, the United Kingdom, and Sweden. The company also offers other financial services through subsidiaries such as Mr. Payroll check cashing centers.
Horne cut his IT teeth on mainframe computers where he learned the importance of physically isolating his computer systems from any dependence on other building infrastructures.
“I refer to our plan as a disaster survival plan instead of a disaster recovery plan,” Horne said. “We don’t want to have to recover from a disaster, we wanted to survive it—and we did.”
The tornado almost completely destroyed the nine-story Cash America Building at 1600 W. Seventh St., he said, but the computer room on the second floor of the building withstood the forces of nature.
The backup plan
Cash America’s disaster survival plan was developed when the company moved into its building in 1991. When company President and CEO Dan Feehan asked Horne what he wanted in a building, he told him he wanted a diesel generator.
“To me it was just a given; any disaster plan includes [a generator] as step one or you don’t have a disaster plan,” Horne said.
“It was something I knew that we needed from the mainframe days, and when we bought this building from the [Resolution Trust Corporation], and it already had it in there, it was just icing on the cake,” he said. “All the company had to do was overhaul the generator and test it every two months to make sure it would stay in operating condition.
“We frequently have ice storms that put us without outside power, in some cases, for a day or two,” Horne said. “So that diesel powered generator was designed to give us power for three days without refueling. That part of the plan wasn’t abstract, because we have ice storms.
“As far as a tornado hitting you, that’s abstract,” he said.
When the tornado hit, the building lost power. But the generator kicked in right on schedule, powering the company’s IBM RS/6000 H50 and older R40 servers until Horne’s people could get to the machines. When help arrived, the first order of business was to mark the machines’ wires and setup in their racks and move them down three flights of stairs to the basement and then to an alternate facility the company owns about five miles away.
“One surprise was that the components themselves were not heavy, but the power supplies were in the racks, and they were [heavy],” Horne said about the 3- by 5-foot tall racks.
The other big surprise from the storm was that ceiling tiles were knocked into the computer center’s air conditioning unit, causing it to fail.
“Fortunately, it was cool until we could take the system down and move it to the backup facility,” Horne said.
Since the tornado, about 80 percent of the company’s workforce has joined Horne’s IT/MIS staff in the alternate facility, a warehouse owned by the company. The other 20 percent are now working in other company properties in the area or are telecommuting.
From things that went right to those that went wrong, Cash America and Union Pacific Resources were able to survive a disaster while keeping their IT functions operational. A few of the lessons anyone can learn from their experiences are:
- Ensure you will have electrical power to meet short- or long-term needs.
- Design your computer center to be reinforced and independent from the rest of the building.
- Have an alternate site from which your company can operate and make sure it has the capability to connect to your computer network—directly or remotely.
- Have off-site data backup.
- Consider a contract with a disaster recovery service.
- Set recovery and business resumption priorities before a disaster.
- Maintain and continually update an emergency contact list.
People made the difference
On March 28, about 14 blocks to the west of the Cash America International Building, winds tore out the windows of the 20-story Union Pacific Resources (UPR) Plaza building at 777 W. Main St.
These are busy times for UPR. Just days after the tornado hit, Union Pacific Resources Group, an independent natural gas and oil exploration and production company, agreed April 3 to merge with Anadarko Petroleum Company.
Obviously, their disaster recovery plan worked flawlessly.
The company’s computer center supports a distributed technology environment comprised of UNIX, Windows NT, and Novell NetWare networks running a variety of business applications, most of which are Oracle-based. The only exception is its financial system, which resides on a Unidata database.
The storm has scattered UPR’s workforce between the original building; an alternate communications center under construction; a work area provided by Comdisco Inc., the company’s disaster recovery provider; and individuals that are telecommuting.
“I definitely consider our activities a success story,” said Todd Coates, IT manager for corporate systems security, standards, and training. He is also in charge of overall disaster coordination.
“At this point we’re still running our disaster recovery plan. I feel very confident that we’re well on our way to success,” Coates said.
First things first
Just after the tornado hit, however, all his energies were directed at enacting the plan as safely as possible.
After contacting Comdisco Inc., Coates contacted the primary response team and determined the status of the facility and the on-site computer operator, David King.
Once the immediate storm threat was over, three other staff members [including Coates] rushed to join King.
“We had other response members on the way; however, the police department quickly closed the downtown area,” Coates said. The on-site response team established a command center, made quick evaluations, and continued executing the plan.
“The on-site team would not leave due the possibility [they would] not be allowed to re-enter the damaged area,” Coates said. “We didn’t want to take any chances, once we restored operations, that something else would happen that we couldn’t react to.”
The re-building process
Now, access to the building has been eased, and reconstruction of the building is well under way. Coates said the company hopes to be able to return to their building in the next week or two; meanwhile, the computer center there continues to function.
Among the things that worked well was that the plan considered the possibility of a tornado and because of that, the company installed plywood walls along the perimeter of the computing center to protect against flying glass and debris.
On the other side of the coin, the things they plan to do better next time include establishing priorities for the business resumption process, Coates said.
Also, after the storm, communications were difficult, he said. Not only that, some of the technology team’s emergency contact information had not been kept current, and although the company’s personnel department was able to provide better information, Coates plans to address that in a revised plan.
“The heroes are the UPR employees and technology teams,” Coates said. “There has been incredible support and cooperation from all departments and all the people involved.”
How well do you think your company’s recovery plan would work? Is everyone who should know about the plan prepared to use it? What other important elements do you think a plan should include? Post a message below or send us your comments.