Networking

9/11: A lesson in crisis control

No IT leader could have been prepared for a crisis like the one that hit New York City on September 11. But one CIO shares some valuable lessons she learned from that event about how to protect data, and lives, when disaster strikes.


Sept. 11 began as an ordinary Tuesday for Leslie Hunt, CIO of the Greater New York chapter of the Red Cross.

As the sun rose over the buildings to the east, she was getting ready for work in her apartment just a few blocks away from her Amsterdam Avenue office on the upper west side of Manhattan. She had just finished her morning cup of coffee when her mind turned to technology. On that brilliantly bright morning—the kind of morning that makes you wish you could play hooky and spend the day on the beach—Hunt was thinking about improving her organization's weakening local area networks (LANs) and wide area networks (WANs), implementing a new customer relationship management (CRM) system, and mapping out a new strategic IT plan.



Then it happened.

“I was looking in the mirror, and I could see the reflection of the television,” she recalled. “I saw the plane hit the building, and I thought, ‘Why are they showing Towering Inferno so early in the morning?’ I walked toward the TV and heard what happened. I thought, ‘Oh my God.’”

What followed was what Hunt described as the most difficult day of her career and some of the most exhausting, yet rewarding, weeks of her life.

First response
Hunt began her work at the Red Cross in 1999 as the chapter’s first-ever CIO. That fateful morning in September, however, as Hunt related, nobody performed the tech jobs they were hired to do.

As soon as she heard the news, she threw on a red blazer and ran over to the Red Cross office to pitch in. By the time she arrived, the scene was already frantic. Civilian volunteers were lining up in droves, clamoring to help and donate blood. Corporate donors kept stopping by, offering to contribute money, computer parts, and just about anything else one could imagine, she said.

As she stood in the middle of the chaos, people flocked to her for answers. By early afternoon, she had arranged for the manning of shelters, made sure food was being prepared, set up a bin for clothing donations, and directed volunteers into manageable lines.

“Those first few hours, I didn’t even think about technology,” she said. “I just wanted to do whatever I could to make everyone’s lives a little easier.”

Finally, around sundown, Hunt was able to shift her focus to IT work. Her first task was establishing an emergency operations center, and she set up 12 desktop computers, a dozen phones, and a handful of network connections for remote Red Cross workers scheduled to arrive that evening. Next, she organized a group of staff members to maintain the organization’s fragile LAN and WAN. She then directed her help-desk supervisor to handle usual user issues, while she focused on making sure the e-mail servers were functioning properly.

For a while, the servers held strong, but into the evening hours, as more and more volunteers came and logged in to the network, stability became nearly impossible as the servers buckled from the increased load. The Greater New York Web site crashed several times, and to make matters worse, the e-mail server got hit with a virus the next day.

“I honestly thought it was the end of the world [when the virus hit]. It was like, ‘Things can’t get any worse,’” she said. “I knew that focusing on the positives, one step at a time, was the only way to recover. So that’s what I did.”

Moving forward
Hunt expunged the virus, patched the hole, and gathered her staff of 11 to assess network failures during the next week. She asked the group to identify the breakdowns, categorize them as procedural or physical, then, finally, led them in a brainstorming session for remedies. A few days passed, and when the IT team completed the task, they had a two-year project plan to improve network security, beef up e-mail servers, replace routers and switches, and supplant the organization’s disaster recovery plan by adding something it had been missing for years: redundancy.

Projects like these cost money, and for a nonprofit such as the Red Cross, even modest plans can be challenging.

“We’re working with limited funding, so we have to be somewhat creative,” explained Hunt. “Our concept is that if it comes between buying a server and helping people, there is no question—[we] help the people.”

So it’s little wonder that Hunt was overjoyed when members of the local IT community began donating much-needed services and hardware to make the team’s project plan a reality. Contributions from AT&T helped make the WAN upgrade happen, while Cisco donated new routers and switches. What’s more, a number of consulting firms offered advice on the e-mail end, and several high-profile members of New York’s IT industry provided expertise on the issue of redundancy.

Today, the plan is partially complete. The new routers and switches are in place, and the new WAN is operating flawlessly. New e-mail servers are on Hunt's Christmas wish list, but she said that establishing colocation and redundancy is likely the top priority next year. She hopes to set up mirror sites in neighboring counties, farther up the Hudson River, by January 2003.

Lessons learned
As easy as it is for Hunt to look ahead, she also has no trouble looking back. She said she relied upon problem-solving skills she learned as a programmer to handle the 9/11 crisis, and she adds that without the “can-do” attitude of one IT troubleshooter, she’s not sure she could have acted so calmly in the hours and days after the terrorist event.

Maybe the biggest surprise, from an IT and organizational perspective, was that the circumstances forced her to abandon the organization’s longstanding disaster-recovery plan. But a new plan will emerge, she noted, as her team will begin tracing how things were handled to prepare a newer, stronger disaster-recovery plan for the future.

Hunt said 9/11 taught her more about the importance of IT than any experience she’s ever had.

“[That day] reinforced the need to have systems in place that enable the Red Cross to free up staff and volunteers to do their jobs without having to worry about the technology they’re using,” she said. “We need to support the organization as a whole, under even the most demanding of circumstances. At the end of the day, computers can’t give hugs; only humans can.”

Are you prepared for disaster?
Write us and explain what your enterprise has put in place to be as prepared as possible for the unknown. We'll share your stories and ideas in an upcoming article.

 

Editor's Picks

Free Newsletters, In your Inbox