Economic business models have traditionally focused on
supply and demand. And while this is a long-respected approach, I think it’s
time to consider a new model based on downtime and money. Somehow, many
companies still aren’t getting the message that modern business depends on
technology—particularly when it comes to communication.

In my experience, many corporations seem to think that the
IT department is the best place to focus their cost-savings efforts. Of course,
they are dead wrong. Technology runs the corporate machine, and it’s time to
adjust traditional models of corporate economics to account for technology
costs.

Many corporations have stretched their IT staff and budgets so
thin that it interferes with the department’s abilities to support the
corporate enterprise—much less keep it secure. But when a malfunction strikes a
critical system, it’s rather amazing how quickly the powers-that-be forget the
word budget.

Of course, you know and I know that technology departments
require adequate funding to function properly—that’s not the problem. How do
you convince the ones holding the purse strings? Here’s a cautionary tale to
share.

About a month ago, a Fortune 500 company encountered a
worst-case system failure. Its e-mail server crashed—the result of a
combination of bad hardware, corrupt data, and e-mail worms. The entire e-mail
system came to a grinding halt, and there was no backup system to bring online.

So, while the IT staff scrambled to get the e-mail system
operational, work throughout the entire company came to a standstill. As e-mail
delivery failed, customers began calling in—resulting in a brand-new problem.
With incoming phone lines jammed to capacity from customers, the system dropped
or failed to complete calls.

The massive call volume also made it difficult for employees
to get an outbound line or use fax machines. The voice mail system was yet
another casualty of the e-mail server problem; it didn’t have the ability to
process all of the calls coming in—or even allow employees to pick up their
voice mail from customers.

While the IT staff focused its efforts entirely on finding
out the cause of the problem and getting the e-mail system operational as fast
as possible, the company’s management was busy rushing to blame someone.
Questions on how this problem occurred were the first to crop up and led directly
to questions about the IT department’s capability.

Of course, the IT staff was well aware of the possibility of
such a problem. However, because management hadn’t seen IT as a “profit
center,” a redundant e-mail system wasn’t in the budget.

Fortunately, the e-mail server stored its data on a Fibre Channel
RAID array. Unfortunately, the failed hardware turned out to be the Fibre Channel
controller, which the IT staff had to order. Two days after the initial e-mail
system crash, the company told employees to take days off as everyone waited
for the necessary hardware to arrive.

During that time, the IT staff struggled to justify its existence,
as work at the entire company ground to a halt. More than a few IT employees simply
quit—both from abuse and from working excessive hours without overtime.

When the Fibre Channel adaptor arrived during day two of the
outage, the IT staff quickly discovered that it would need to restore the
entire system from backup due to data corruption. With more than a terabyte of
data to restore, it was evident this wouldn’t be a quick process. As a result, the
finger-pointing continued.

The IT staff worked around the clock to bring the e-mail
system back online and restore the data, with its efforts culminating in success
on day four. But within an hour, both the e-mail system and the Internet were
unusable again—the company didn’t have enough Internet bandwidth to begin with.

While the e-mail server was down, management instructed the
IT staff to open access on the firewall so employees could use free Web-based e-mail
services. Open access to the Internet led to more than a few incidents of
viruses, spyware, and more e-mail worms. In addition, a lot of existing viruses
and spyware were present on computers, few of which even had desktop virus
protection.

After disconnecting infected systems and a few tense hours,
the IT staff managed to somewhat recover the e-mail server, and the Internet
connection was no longer flooded with incoming SMTP traffic. The Internet connection
was reportedly “slow as always” but usable.

This is about the point where I came in. Management asked me
what they could do to prevent this from happening again. I told them to invest
in the IT department in order to replace the employees who quit in disgust, purchase
a backup e-mail server system, and increase Internet bandwidth.

While I can’t tell you exactly what the incident cost this
company, I can tell you that it would have cost much less to prevent it in the
first place. In this case, to avoid almost a full work week of downtime, this
Fortune 500 company could have spent approximately $25,000.

Without a doubt, this system failure cost the company much
more than that—not just in dollars, but also in the loss of intellectual
knowledge as well as customer satisfaction and trust. Even worse, this entire
situation was preventable had the powers-that-be understood that you can’t rely
on traditional economic models to dictate technology investments.

Miss an issue?

Check out the Internet Security Focus
Archive, and catch up on the most recent editions of Jonathan Yarden’s
column.

Want more advice for
locking down your network? Stay on top of the latest security issues and
industry trends by automatically
signing up for our free Internet Security Focus newsletter, delivered each
Monday.

Jonathan Yarden is the
senior UNIX system administrator, network security manager, and senior software
architect for a regional ISP.