IT leaders are good at testing systems and policies, but often overlook real-world scenarios. Here are emergency situations every IT department should train for.
Most IT organizations would consider themselves competent in testing. They have decades of experience, a well-defined methodology, and modern testing tools. However, few test the chain of command and organizational ability to respond to challenging incidents.
Recently the United States government performed a simulation of a large-scale cyberattack on U.S. infrastructure. As one would expect with this type of attack, primary targets included civilian infrastructure like the electrical grid and financial institutions, in addition to military targets. One of the surprising findings was that the military does not have a clear organizational way of responding to attacks on domestic targets. In a real incident, valuable time would be lost as military and domestic entities attempted to coordinate responsibilities and responses, ultimately giving the attackers more time to inflict damage.
Presumably, the U.S. has the latest technology and training for responding to large-scale cyberattacks, yet in this case organizational problems prevented a coordinated response. Whether you are leading a complex military organization or a small IT team, the human element is key in responding to crises, even crises far less dramatic and threatening than a multi-front cyberwar.
IT can learn from emergency management
Since we, in the non-military world, generally don't wear badges of rank on our shoulder, and in many cases have abandoned hierarchical management structures, understanding and testing these human systems and chains of command is perhaps more challenging, Worse yet, there may be challenges embedded deep within your organizational culture that can instantly derail a response to even a mundane problem. In organizations where minor failure is punished to the extreme, many nominal leaders and managers will sit on their hands rather than risk making an incorrect decision in a time of crisis. Identifying and understanding these challenges is key to determining a solution, and rarely is an effective solution as simple as buying some new software or technology.
Test your organization, not just your systems
While most technology testing is focused around systems, interfaces, and processes, testing your chain of command should be focused on scenarios. To test the government's response to a cyberattack, a multi-front attack against military and civilian targets made perfect strategic sense, and exposed the weakness in a chain of command that separated these areas and delayed a response.
Similarly, testing for flaws in your organization should not focus on specific systems or processes, but on sensible scenarios that could occur in the real world, such as the following:
Serious vendor vulnerability: Rather than considering what would happen if your ERP system failed, consider what would happen if your core enterprise software vendor discovered a vulnerability that affected all their packages.
Major web site rollback: Rather than testing how you would "cutover" to a new customer-facing web site, test how you would respond to a demand for an immediate rollback to the old site.
Social engineering attack: In addition to elaborate, technically-oriented security testing, call 10 non-employees and ask for their passwords while masquerading as the help desk.
Insider data theft: What happens when someone calls the help desk to report an unfamiliar person who seems to be downloading confidential customer or product data in the cube next door? Will a response be organized in the few minutes it would take to complete a major data theft?
Critical change request: What occurs when a major customer rings her friend in marketing and demands an immediate change to an IT-delivered product?
Continuity of IT leadership: What if you're backpacking miles from a cell tower, and there's a critical failure that has the CEO storming into the IT cube farm demanding answers?
Some people refer to these types of incidents as "fire drills," but that might be inappropriate since fire drills are often practiced and rehearsed multiple times. For organizational challenges that no one took the time to conceive and test, respondents are usually making up the response process as they go along.
A flawed organization can derail even the most thoroughly vetted technologies and skilled staffers. As soon as a response is no longer coordinated and planned, chaos gradually overthrows an orderly and successful response. Whether you're exercising your response to a security incident or trying to determine how your organization would respond to a major customer complaint, testing your organization is every bit as important as testing the technology.
Former NSA and CIA director recommends managing consequences instead of vulnerabilities
Hurricane preparedness gets high tech in Fort Myers, Florida
11+ security questions to consider during an IT risk assessment
4 tips for disaster recovery communications