Discussion Locked

Why are heads NOT rolling?

0 helpful
193 replies
Closed to new repliesThis discussion is closed. It stays readable, but it takes no new replies.

Am I the only one here that thinks this, but why are heads not rolling?

Many large virus / worm outbreaks in the last several years that have afflicted many large companies have almost ALL been preventable.

The infections have been avoidable either with proper Firewall Protection alone, or proper Anti-Virus Protection alone, or Proper OS Patches and Updates alone. With all three protection systems in place (ALL PROPERLY CONFIGURED AND UPDATED!) these viruses would not have infected most systems.

So, if these infections were all completely preventable, why are IS and IT heads not rolling. Why are these people NOT being fired for incompetence??

If they requested that technologies be in place or implemented but approval from above was refused, then why are the VP or CEO heads not rolling??

Some staff members can make a simple mistake that costs a company a bit of money (like a user not following a policy, infecting their own system with a virus, and they get fired for it). Yet, these preventable virus / worm outbreaks cost some corporations millions of dollars in down time and lost business, yet no one gets fired?

What?s up with that??

193 replies

by jtrainer

In my organization, we have to keep our version of Windows 2000 Server at SP2 and SQL Server 2000 at SP2 because of version compatibility with our ERP. Change the SP and the app no longer works. Upper Management refuses to pay the maintenance fees so that we can upgrade to the lastest and greatest. We are vulnerable to attacks because MS doesnt support those earlier SP's with security patches. When asked to update subscriptions to our anti-virus software, the response is "tell them not to use the Internet." I think upper management needs a reality check and realize that to prevent these outbreaks you need to free up some capital.

by rloski

Management needs to make sure that their IT personnel have the time to keep up to date on threats and their solutions. We can't be working at 100% fixing problems and have time or energy to research potential threats.

And that means money!

by ali40961

How many times has our org had to jump thru hoops trying to make apps work with the latest and *greatest* patch/SP? Too many to count. I would b a rich woman if I had a PENNY for each incident.

by trekman1

Anyone who suggests to use windoze over a Unix OS (SUN, MAC OS
X) in their environmnet should be fired

by richards_unsubcribe

Upper management is generally non-technical and they often don?t know what happens in the server room. It's pretty easy for the IT pros to pass off problems, security vulnerabilities, as problem created by someone else. The day-to-day operation of many companies is highly dependent on the expertise of the IT staff. Upper management, bean counters, and the other sundry pooh-bahs often don't know or even want to understand what's happening downstairs. It's pretty easy for an IT pro... who the company is highly dependent on, to pass the buck.

But let a scribe... let a lowly cubicle dweller screw up and an example must be made to all... its summery dismissal for "violating" a company policy!

This issue is more about office politics and the corporate pecking order than it is about "security". It's about who counts and who doesn't, who can pass the buck and who can?t, who is expendable and who isn?t.

Someone tell me I'm wrong...

by bfilmfan

Becuase the impact to the business hasn't affected their bottom line.

When it does, management is out on its ear and new management bozos will replace them.

by fcleroux

What? Are you out of your mind?

As I mentioned, millions of dollars.

I many cases it cost hundreds of thousands alone in overtime or consulting fees to resolve the problems but this amount is small in contrast to the MILLIONS cost in lost revenues because of down time! Not to mention the lost man hours because of the thousands of employees that are then sitting at their desks not doing any work for several hours to several days.

How could you not think the bottom line is not affected when several thousand computers are out of commision for half a day or longer.

by jdmercha

The problem is that the bean counters can only count solid objects, desks, chairs, servers people, etc. The empoyess charge there time to a project or department. There is no account for 'fixing that which could have been prevented'. The bean counters can't see revenue lost due to wasting time. Thus, if the bottom line is not looking good you have to cut an object, not wasted time.

by bfilmfan

I won't argue your point in the least, as I agree in principle.

However, as other peers have pointed out, there is no accepted accounting method which allows for "Business Lost Due To OuR Not Getting It" for whatever reason.

You don't get the contract cause the network is down, you don't get the contract cause the salesman is a bozo. Both of these have the same negative impact on a business, but neither can be counted as a loss, but is simply business which never happened. And this is how most businesses view and deal with computer downtime.

I think it is a stupid manner of doing business, but I recognize that it is.

by lordshipmayhem

Because it's not OBVIOUSLY affecting their bottom line.

As soon as the systems are up and running again, nobody bothers to calculate the cost of the latest system outage.

If they did, then bosses would be screaming for something more secure.

by bfilmfan

Very few organizations are able to calculate the cost of "downtime," as they have no idea what their "uptime" revenue stream is either.

And I agree totally that they should calculate that cost as it is a business loss, even if they choose to ignore it.

by dr dij

IT guys often ask for corporate wide anti-spyware or AV, or adequate firewalls, etc and get turned down. Until recently we didn't have corporate AV solution and only got spam appliance recently.

management also often understaffs these depts, so the people would not have enuf time to protect their networks possibly.

So much of the time, management needs to hold the mirror on themselves. Which can be a hard concept for them as they often do not understand IT adequately, treat it as a loss area.

by fcleroux

That is my point. If middle management saw no need for an AV sollution and yet that decision cost millions of dollars to the company, why are they not being fired.

If they developed a new product that was a bust, they would get fired. If the made a mistake in product development that cost millions they would get fired. If they made a financial mistake that cost millions they would get fired.

Why do they NOT get fired for making an un-informed bad IT decision. It is their job to make sure the systems are safe.

Billions are spent every year making sure systems are available 24x7x365 with no downtime. Redundant everything, clusters, mirros, backup systems, yet they do not want to sped the little extra to make sure a patch is applied or that AV updates are done. What is the cost of one or two extra IT people compare to the millions already invested in hardware and software alone?

by peter spande

I'm guessing if there are individuals saying no to corporate wide AV or Anti-spyware solutions it is senior management (perhaps finance, operations, etc.) and not IT.

I could envision a scenario where it is the administrator that didn't catch a patch update that is in jeopardy of losing his/her job and not the senior managers... however unfair that might be.

by firewalker91

We are a production environment that runs 24/7/365 with rotating shifts to cover the time. We manufacture products for Ford, GM, Chrysler and some other auto companies. Any downtime means missed shipments and a big auto maker putting pressure on us threatening to switch suppliers. Because of the pressure from our customers not to miss shipments, we've had to create workarounds not to stop the production lines.

In the office, it's easy to set up SUS, WSUS, or whatever program to push security and anti-virus updates to the clients. That's a no brainer and someone should get fired if they can't set up a simple tool like that (and it's free).

The production environment is different. Because an anti-virus program scans each file as it opens and closes and our testers can create or access dozens of files for each module being tested, it slows down the tests and doubles the test time (less products produced = less products shipped = missed shipments/unhappy customers = no new business = less revenue).

Also, most of the applications that do the testing are custom made apps that still run on NT 4. Adding a service pack or security patch has actually prevented the testing programs from working. Since the applications are older, some of the companies that wrote the custom apps are no longer in business, so we have no support if something goes wrong after adding a patch (if something goes wrong then: less products produced = less products shipped = missed shipments/unhappy customers = no new business = less revenue). We've had to do workarounds on the testers such as shuuting down extra services, running user accounts with READ-ONLY access to other computers, etc., in order to minimize risk until we can spend the millions of dollars to replace the equipment and the months it takes to verify and prove out the new equipment. And the cost of replacing the old equipment is a lot greater than the cost of a few hours of downtime if a virus hits.

by shorne

This is a bit off topic but another angle on this is that if your middle or upper management are refusing to fund a decent corporate AV solution, it's probably not a great place to be and you might want to move on before the S*** hits the fan. If they can't even commit to basic technology like this they likely aren't supportive on many other IT projects, unless perhaps it involves a toy for them.

by mcollins1

I thin that's the problem. Managers should be made to go on a course to make it hit home how important AV and Anti-Spyware is. It is wrong to place the finger of blame at the IT staff, when it is possible they are doing their job to the best of their ability with limited means available to them.
I know that my company has only very recently recognised the risk of Spyware/ID theft etc. And have only very recently put in measures against them, despite being told for years!

by 2wired

Many ppl also fail to factor in the loss of earnings resulting from lost revenue from potential customers, or indeed current customers, who pull out or choose a competitor as you?ve earned the name of being susceptible to virus attacks. This could, for example, occur if some worm got hold of a customer email list and mailed itself onto those companies networks...spreading itself and infecting these companies too.... or even worse, if someone gets hold of customer bank/credit card details! In effect then you?re now responsible for your customers get hit.

This happens once or twice and you will find you customers pulling out left right and centre. After all, if you cant be trusted to update your own anti-virus or provide adequate security, why should they trust you to deliver your end of a multi-million dollar contract? And ppl talk, so its not just existing customers that will be affected, but potential future ones too. Word of mouth can greatly affect ones customer base in ecommerce or other such areas when all many companies have to rely on is their good name.

by cwyman

I'm not trying to attack anyone here, but I thought that I would add my recent experience with what I consider to be accountability.

We reside in a network where users bring in their own equipment and hopefully abide by policies that we have developed, (ie. having anti-virus, spyware protection, software firewalls, etc.) However, this is only policy. I have dreamed about holding a user accountable in a monetary fashion for several years since it's usually ME that spends the countless hours cleaning up the mess. IT mgmt feels the same as we, but no one took the time to develop a way to place a dollar amount on an outage.

Well, it was time and so I stepped up to the challenge. Now granted, I'm a little one sided, but we had to start somewhere.

I've recently implemented an incident report template to be used whenever a virus attack, nw outage due to neglegance, theft, etc., occurs.

The last section is Costs of lost opportunities. When I first developed the draft and presented it to non-IT mgmt, they of course got very upset because their claim is that you can't put a dollar amount on something that is not set in stone. They of course are referring to the fact that our users work on a commission basis and that varies per person so it's not like....okay, we've got john doe here and he makes $30/hr and his machine was infected and total cleanup took 8 hrs so that's $30*8. I did have my numbers based on an average and basically the numbers were right. But, because the particular case study that I demonstrated showed a lost of $2.8Mil per hour for 2 hours was devastating to them (hence the importance of accountability I'm trying to bring to light!!), they didn't want to go that route. Now granted, I see their side of the picture. They feel that if we start charging back to those that allowed the incident to occur, then we put ourselves in a he said, she said attack mode and that's just not good business. So the next time that we cause an outage because of a programming error, they will try to charge that $2.5mil back to us (which is understandable).

So, we went back to the drawing board and with IT CIO we were able to make a compromise. We used a percentage to at least represent some type of opportunity costs lost. Now we did have some definite dollar amount charges (We have 3 sections - HO costs for man hours and loss of Internet costs, - Costs for the individuals that had to be down, - Costs for larger outages), but we were able to give representation to upper-management that these are serious and we need their support to help educate the users about the responsibility on their end, not just ours.

My point I guess is that, it's painful to hear the truth and so they try to avoid it, this is true, but if you're committed to making things better than it's up to you to work with others to get the message across even if it means working together/compromising without sacrificing to come to a resolution that works.

I definitely feel that even though this incident report will not make people happy, it will help management come around and begin supporting the IT department that so desparately needs their support in order to be able to function as needed.

This is what we've had to do start on the right track....

I do agree that blatent (sp?) neglect of policy, should ABSOLUTELY be cause for reprimand and then dismissal.

Put simply...if I tell you not to touch the HOT pan of boiling water on the stove and you go right over and touch it, then I have no simpathy for you when you burn your hand. And then if you
re dumb enough to do it again, then you get what you deserve.

by Member 30634329

I don't think that you can be so quick to start firing IT people when an outbreak occurs.

With public companies, all changes to appliance and application configuration must go through a formal corporate review process and completely documented. Adequate backups must be verified and proper change management procedures followed. This slows things down incredibly.

Also, patching and other changes are far from automatable. Changes must be put through a testing process to make sure they don't break applications and IT services. IT personnel can't fully test every application....often we have to wait for users to verify proper operation in a test environment. I've seen more downtime due to not testing patches thoroughly enough than I have for viruses and worms.

I disagree with your statement that all viruses and worms are preventable. Even if your company has all of the best systems in place to prevent this from happening, nothing is 100% secure. Your company is still vulnerable to attack!

You state that staff members can make a simple mistake and get fired for it. You can't apply a slippery slope here, because a virus outbreak is often not a simple mistake. Every company I've ever worked for treats virus outbreaks as an opportunity to learn and grow....instead of an opportunity to fire people. Following your logic, wouldn't you also fire the person that unknowingly opened a virus-laden e-mail attachment from "System Administrator" to start the whole thing? Or the user who gave their password to someone pretending to be from IT?

by fcleroux

Several Point here...

1) I did not state that ALL Viruses are preventable but rather that MOST are.

2) I did not suggest that everyone at all companies should be fired if there is a Virus outbreak. Rather I just asked the question as to why people are not fired. There have been many large outbreaks in the last several years, why has NO ONE been fired (that we know of). Perhaps no one was at fault at all?? Doubt it! From my own experiences within the corporate world I know of several MIS people that should be fired for what I would call negligence.

3) "Following your logic, wouldn't you also fire the person that unknowingly opened a virus-laden e-mail attachment from "System Administrator" to start the whole thing? Or the user who gave their password to someone pretending to be from IT?" - I would not, but yes people have been fired for such things.

by dmambo

We got hit by Zotob at a couple of other corporate sites. When I asked why those sites weren't up to date with patches, the grapevine told me that the guys responsible for those sites hadn't applied group policies to new machines for a couple of months. When I got the official corporate explanation, it was that Zotob was so new and so voracious, it was fortunate that it didn't bring down the whole network. It was clear to me that there was plenty of a$$-covering going on, and the higher-ups just didn't know the right questions to ask.

Nobody in IT is going to say anything either because it just as easily could have happened in my backyard or anyone else's. It's like cops not squealing on other cops.

by jmgarvin

Ya, that is a big part of the problem. Another part is that management sometimes "stops" patch rollout with their "test and comment" syndrome. Sure, testing is good, but deployment needs to happen quickly and quietly (usually in under 48 hours and usually while users are working because there is no good time.)

A major beef of mine is that nobody is accountable and the attitute is that sheisse happens.

by jamesrl

In the 90s I was part of an anti-virus/security for a worldwide organization. When Michaelangelo threatened to wipe out PC HDs, we all took precautions. But inevitably some people got hit, albeit a small percentage. And I spoke to a few people at other companies who had similar experiences.

Yet when interviewed, our PR types denied any knowledge of any impact.

No one wants to admit they were caught with their pants down. Its embarassing. So collectively it seems the whole company wants to downplay it.

James

by jdclyde

Because they came along and "saved the day".

This goes back to why many Windows Admins get more respect than Unix Admins. The powers that be don't understand or CARE what it takes to make something work, they just know they SEE you doing something all the time.

Many Unix servers are just forgotten about until it is time for an update or patch.

Many Windows servers will need to be rebooted because of memory handling issues.

Which SEEMS like the better worker to non-techs? The one that comes and saves them all the time.

Getting a virus and then having your tech clear it makes you NEED that tech, not wonder why you got the virus in the first place.

The media also helps with this over dumbing down of the execs on just what we do.

by stress junkie

When I've run VMS machines it's not unusual for people to ask me what exactly I do. They don't see me monitoring resources, getting to know how people use the machines, and coming in on weekends for performance tuning and preventive maintenance. I have taken people asking me what I do to mean that I'm doing a good job. If my work was visible to them it would be because it was causing problems for them. I think you're right about the Windows admins getting more visibility and therefore more credit. Windows machines always have patches to roll out, problems from patches being installed, virus infections. Nontechnical people think that this is normal and that the Windows admins are doing a good job.

by jmgarvin

The media is making matters far worse with their inane ramblings and non-technical "in depth reports."

Only Windows is effected, so that means that it is better! Oh and Bill Gates loves you!

by grbeckmeyer

to not have any Windows servers here. So what's this 'Zotob' everybody's talking about lately?;)

by bluegiant

I'm in a similar situation here too. I administer an iSeries (AS/400) running OS400 and the ERP app that resides on it (JDEWorld...or whatever it's called now ;) ). Very little recognition because the thing just runs and runs. No viruses, no problems, it just works. Just install the occasional PTF, keep the system cleaned up, and everything's good.

The windows side of the house gets the attention because he is always involved in fixing a problem somewhere. Granted, out Windows admin is very talented and keeps everything up to date and very secure, but as you know, there's always a problem to fix when Windows and basic users are combined.

Now back to the topic...there are many ways that a virus or worm can enter the system. In our situation, it mainly depends on whether we did our due diligence to prevent the virus/worm when disciplinary action is considered. If there were things that we reasonably should have done but failed to do, then disciplinary action will probably happen. If someone brought the virus in as a result of violating company policy, they will likely face disciplinary action. Fortunately, we have not had a virus or worm get to our network in years.

There are a lot of good posts as to why so many other companies have these problems without consequences...incompetence at executive and middle management, corporate politics, and technical ignorance throughout the chain.

by jdclyde

Blaster? Slammer? Just things they talk about on the radio.

And then the ones that DO use a Windows Server, have you ever heard of a DMZ? Isolate your mail and web servers from the rest of your LAN. Is that REALLY a difficult concept?

How about NOT running EVERY service as Administrator?

NOTE, using SAMBA, we have a few Window packages running on Linux servers without a hitch. The software doesn't know or care, it just runs.

by jkaras

They made themselves look bad, it doesnt effect you. I think the last thing we need is more IT people out of work. Just think if the slightest thing happened, whammo your fired!! Sounds good huh? Nothing is 100% effective. Now if those same people got hit again the same way without any precautions, then hello ladder climb! Well, that's my take on it anyways.

by mcollins1

I think it is hard now to find experienced employees in IT, who really know what they're on about. If a firm finds a good employee, they will not necessarily want to sack them over a mistake such as that. The person may be judged more on how they react to, and remediate such a situation, and not on whether they use the correct preventative measures originally... What happens afterwards however is another matter. They should be pro-active in trying to stop it happening again.
However, I know firms who are pretty well protected and still get the occasional virus outbreak. It's a fact of life, and it is a matter of containing it rather than letting it grow too much.

by andeanderson

of what actions have been taken as a result of an infection.

Many corporations prefer to keep their activities from the prying eyes of the media and public and so maintain their image of being clean and in control.

There are many empolyment contracts with separation guidelines which would leave the corporation open to litigation if they announced that Mr./Mrs./Miss. Applegate was removed from employment because they introduced a virus into the companies computer network, either by accident or as an oversight. So, unless there are criminal charges filed the information will never be made public.

"Public Image" is very important and as the Japanese Culture has taught us "Saving Face" is one of the most important activities a man or corporation can accomplish.

by sean_tan

Juz like our immune system, there is no foolproof evidence dat we can take on new viruses or pathogens. Computer viruses were introduced round-the-clock esp. with Internet connecting the whole world together. Most of the times the new viruses infect the companies even before the patches, firewall or anti-viruses are ready. The root of the problems are from those virus-creators, their heads should be chopped instead.

by bob_steel

I couldn't agree more. Basic hygiene and security controls are all
that are needed - if an IT pro can't get that right they should be
encouraged to take up a different career.

by douglasjohnledet

The bigger question is why don't corporations sue Microsoft for "defective" products, i.e. MS Windows and it's known defective security holes.

Doug

by pcteck

Only once can I remember anyone asking, "how much did this virus cost us?" When we told them and it was like, "ok". We suggested what was needed to help lower the risk but the bean counters said "to much money at this time" Do they think it will get cheaper as the threats get more powerful? I agree that as long as it goes away everyone is back to business as usaual. Bigger penalties for people caught writing virus. Doesn't happen very often though.

by plumley9

You can't sue Microsoft or any other 'off the shelf' vendor because the shrink wrap license you purchase (you do know you are not buying software?) includes limits on damages and NO promise of usefulness. For forty years the software vendors have massaged the laws so they are more protected than any other industry in the world. Your ONLY remedy (since the states and feds wimped on 'monopoly') is to STOP using the product. And if you whine loud enough you may get the license price back from a small company never from Microsoft. And technically in most states you can't even sell the license to a 3rd party to recoup your losses.

by rzimmerman

This is the ongoing embarrassment in this industry that software vendors can't warrant their products for "fitness for use". One of the many reasons this industry has a major credibilty gap.

by eliad_08

well all i know is that all heads are rolling and IT's and IS' are doing their job(well most of them). but my point is that, IT's are making a secure firewall, and using a perfect anti-virus, even releasing a memorandum on how to prevent virus /worm outbreaks, but unfortunately these office staffs or even the CEO's are very stubborn and doesn't pay any attention to these warnings, and still they download unwanted attachments, visit PORN sites, etc. why should they fire the IT's if they are doing they're job??? the truth is that, not all firewalls are that secure from virus/worm outbreaks, they cannot block downloading attachments from emails, and some anti-virus can't even detect the newest virus\worm. it shouldnt be just the IT that is doing the prevention, the office staffs and CEO should be doing their share of work on this matter. in order to have an computer that is virus free you should follow these simple steps:

1.) informing their colleagues about a new virus
2.) they should'nt open attachments if they dont expect that or if its from an unknown sender.
3.) stop visiting porn sites coz some of them has virus/worms.
4.) always update your anti-virus from the internet so that your anti-virus should be able to detect the latest virus/worm.

by brain

I was able to attend a "Certified Ethical Hacking" class to get an understanding of how trojans, worms and vulnerabilities can be attacked, and I tell you now that the only way!!, ONLY way to prevent your self from getting a virus or to prevent getting attacked is to keep your computer totally off of EVERY network. No internet access, no network connectivity to any other device, no floppies, no cd's or media that can be inserted into it, no phone lines into the computer.
Firewalls are great tools, but all that does is slow someone down, it will not prevent someone who is determined on getting into your network.
Reality says that that isn't possible. Since that isnt going to work, you still need to have firewalls in place, both hardware and software version running, a constantly updating Anti-Virus software running. applying patches and service packs are a necessity - and need to be applied religiously (a very tough thing to get agreement on in management becuase of the potential to break home grown or old software). Tightly controlled Internet browsing software that is managed and maintained and using high levels of security should also be in place...
there are many more things that could probably go onto the list... but End-User training can go a long way, if they know what is safe and not safe to do it may make our poor jobs alittle easier...
i have not finished ranting..
have a good day.

by deadly ernest

refused to fund the required equipment / software / work do not have to account for the non-funding to the share holders. Everytime I have seen avoidable damage done to a system it has been because higher ups have refused to pay for the needed work, in most cases the IT people kept their requests and refusals and thus covered their rear ends, and the people responsible were to high to get shafted for their stuff up.

by blastfurnace

My head should roll IF I have the ability and authority to do what I need to protect the network.
I ran a network for years before anti-virus was bought..... I do not hold the checkbook.

by the monkey

perhaps they are doing it but getting the calculations wrong, most businesses see the cost of mantaining a proactive it department( having the staff and resources to deal with whatever would cause the downtime ) greater than there perceived costs of the downtime

by cerisew

Simple math

Take a $3mil annual revenue, divide by 52 weeks and you get approx $57693 per week. 8 hour workdays 5 days a week, about $1443 per hour. Figure you lose a percentage of that when the network is down (not everything grinds to a halt), we'll be conservative and say 35% of the revenue is gone. That means it costs the company about $504 per hour if they have a buisnesswide outage. If it's a local outage, say the buisness has multiple locations, the downtime could drop less than 10% of the profit. So the larger the buisness, the less intrest they have in the administrators.

by gitmo

If a company can calculate an ROI to justify their IS investments, then they can calculate a loss due to IS resource unavailability.

I think most large companies delivered a reprimand at the first noticeable outage. A second preventable outage would result in a dismissal.

by bob

It seems as if some people come up through the ranks of corporate IT and then forget their past.
They don't seem to remember how IT's hands are tied, that there are many aspects of the job that management calls the shots, no matter how much recommending, begging, and pleading is done.
Sure, fire all the IT. Then let those management types keep the patches, updates, and firewalls in place.
How long do you think that will keep companies in business?

by consultant-1

The problem may start with Lamers on the IT staff who don't maintain patched systems, but they are enabled by IT management who don't know that the machines should be patched. Then, the CIO's bosses (who don't even know what a cpu is) buy off on some spin by the CIO as to why their productivity was interrupted by a virus, and so the ignorance chain goes on...
But why just look at virus attacks? I have had clents who are still using Win95 and DOS machines, and who had to be told why a more current OS was desirable!! This ignorance was from the techies on a 1000 node network!

by cyber_daddy

First of all i will applaud your objectivity on this matter. It is very easy for onlookers to raise the question. But the reality of any security or general admin job in a technology environment is that a component of the job will always be "Reactive". Companies will spend millions of dollars on "proactive" measures to prevent this, and some companies will spend even more money in reseaching these methods. We rely on these companies to advise us on How we need to protect ourselves and only to the extent that they have mechanisms to make us more proactive (i'm thinking patches and virus definitions here), we are only as proactive as they are. And so are the heads we are quick to put blame on....

by it security guy

And there is the fact that it is not the fault of the Admins who don't patch. Operations Management must be on board and give approval. But if their management says no and/or doesn't provide funds to keep up-to-date, then the environment will suffer. It is also true that not all environments can be updated because of the custom code being used, although there are perimeter protection schemes that can be implemented. It should be the top management who is made aware of the consequences of not patching, etc and it should be they who pay the price if they don't allow their managers and staff to stay ahead of, or at least on top of, the latest security developments. Having patching policies that management agrees to is a start.

by mikeholli

fcleroux, let me explain something to you. Virus
control is....iffy at best, when we have to
deal with hundreds or new viruses, trojans
keyloggers, etc. It is NOT possible, even with
keeping your .dat files updates daily. Even with
McAfee's Emergency Responce we can only fight
with the tools we have available. Firewall protection? HA! Enduser signup for all sorts of
crap that get them emails with attached viruses,
etc on them! There is NO real WAY to prevent it
from occurring. We can stick a bandaid on it.
In the form of a Firewall, and Anti-Virus
Protection, but unless McAfee or Symantec have
.dat files for viruses that aren't even here yet
(magically) we cannot stop it from happening,
only try our best to contain and repair it.

Mike

by eric.hayes

Check out Cisco Security Agent.

They market it as a "Day Zero" defense program. It prevents certain types of behavior in an executable, rather than protecting by scanning file names like most AV programs. The idea is that you don't need an updated .DAT file to detect a malicious program. We recently purchased it and are in the process of rolling it out.

Nothing is foolproof, but when AV, FW, and Patches don't seem to be enough.....this product should help.

by fcleroux

Mike, you didn't need to "explain" that. Yes I know that!

You missed the point. If you read some of the replies you would see that several companies just recently got Anti-Virus Protection. Many companies still do not have firewalls, MANY companies do not have Anti-Spyware programs in place.

The question is not about what happens IF you have everthing in place and you still get a dissaster, but rather why is someone not fired if they have not put everything in place.

These companies (as mentioned in some replies) that just recently got Anti Virus sollutions installed - should somone have been fired if they had had a major Virus outbreak before the AV systems were in place. Do you think that an IT manager is doing their JOB if they do not even have a Firewall in place or Anti Virus software installed in this day and age.

by samuel.custer

Any up to midlevel manager that is worth his salt has dossier on everyone in sight, and also it would be a bad reflection on any company to admit that tht uper to mid level managment is populated with people that don't have a clue, but depent on the peons that they sweep out with impunity.

by damunzy

I am pretty proud of how the corporation I work for faired. It is a large corporation with over 100K employees distributed all around the world. It was reported that 95% of our systems were patched and uneffected - not a bad number! 100% would be nice but 95 is nothing to scoff at.

by dmullins

If anyone should already know you don't just throw any/latest updates on your server with out first testing them. I know we had several application that whould only run on NT4 SP3. If we were to update the system the application would crash. It takes time to test updates, while sometimes there is only a small window when a server can be taken offline to do such maintenance, and still sometimes you just can't do it.

by chip_langowski

The competent IT middle manager has learned to keep detailed records when their requests for applying patches or upgrading security are denied. Those records come out when an outbreak occurs and senior management starts looking for someone to blame. Unfortunately, the people with the authority to veto upgrades usually have the authority to cover up their poor decisions.

by fcleroux

I started this Discussion. Read many answers and think you hit this one on the head!

Well done.

by jhogue1

This is the best answer yet. IT is automatically going to scream a lack of money or labor is to blame for the problem. If IT can prove they asked for funding and were turned down then the responsibility should fall on the next level up. If they can't prove that they asked for the money then IT has to take the heat.

by fcleroux

If you read the dicussion you would see that we are not in favour of firing everyone that is involved in a Virus Outbreak.

Managers, IT people that are all doing their best and have taken precautions should not be fired! But what about the few (some comapnies still do not have Anti Virus Software and many more do not Have firewalls and Anti Spyware) that are not doing their jobs?

Would you fire the Bank Guard if he had left the building to go for a beer or was sleeping on the job when the robery took place??

by rope

If these crimes were equated to a home invasion crime with the equivalent pain, suffering and dollar loss the criminals would have a huge number of law enforcement personnel on their tail. If a criminal break?s into your house and you have reasonable security are you to blame for the damage? If this break-in had the dollar amounts that are talked about here there would an international manhunt for the crooks. With the sum of pain, suffering and dollar loss from these crimes, the punishment and law enforcement should be the same as in the home invasion crime. Why is the blame for the crime being place on the lock & door makers and installers or the transportation used to get to the home that the criminal is invading?

by dboundscpa

Excuse me, jdmercha, but you are beyond incorrect in your hypothesis. Being one of those bean counters, I can assure you that counting lost productivity is among the most significant things we bean counters do. Whether you are aware of it or not, if your organization is worth anything, someone is counting the cost of your salary or wage, plus all the additional costs of employing you, and letting management (all the bosses) know how much it costs them when you don't work. The real problem is that management is too damned scared of IT to make significant changes in personnel. When a virus hits the system, so many people have fallen down on their responsibility (typically that same management is at the top of the list) that to remove all those responsible would itself be catastrophic. The managers who believe that overhead costs such as protecting the system are just not worthwhile should be replaced with people who think in terms of reality. Then systems would get protected because management would proactively work to make it happen.

When it's all said and done, please don't blame the bean counters!

by elder griffon

I'm not sure it was jdmercha's intent to blame accountants as much as to observe that accounting as a discipline struggles with reflecting certain kinds of costs.

I really don't know much about the subject at all. I'm sure you're right when you say that part of the point of accounting is to get at those costs which are not at all straightforward, like lost time. But isn't it also true that accounting practices can provide management a means for ignoring inconvenient costs? For example, couldn't a company move expenditures for virus outbreaks to some part of the balance sheet that makes them look like normal, predictable costs? Just asking because I don't know.

Judging from news reports, it certainly seems like certain shady practices occasionally become endemic among companies, like hiding compensation for executives.

by davidives

I recently left my CITO position and returned to academe to teach (a course in Cybersecurity, among others). As I told my class 2 days ago, the fact that no one in IT/IS or in upper management has been fired for their gross incompetance and/or dereliction of duty -- and for putting *our* data and information at risk, at that -- indictaes that ignorance and mismanagement is indeed a powerful force. It looks like all of us had better plan on defending our information and identities for decades to come; there is no indications that corporate or governmental America knows what it is doing, or knows what should be done.

David

by jck

Expecting management to fire IT/network support techs for non-obvious network and computer (OS/application) vulnerabilities would be akin to expecting your automotive repair provider to fire any mechanic who didn't know your car was going to break down ahead of time.

I disagree with your analysis that many of the virus outbreaks being "... having almost ALL been preventable." Most virus hits are not preventable until reported and patched.

I do agree that IT should be held responsible for incompetence. I worked at a multi-practice medical firm where the IS manager was in charge of server and network security. This guy (a former manager at Microsoft in Redmond) didn't even have a firewall on the high-speed link coming into our network. Subsequently, we were infected with Code Red (and other subsequent attacks) long after it'd been patched. He, due to his non-diligence to secure private medical information, should have been relieved of his duties. If HIPAA had been required at that time, he would have put the firm at high-risk for law suits due to the requirements to anonymize and secure private medical information.

However, he wasn't. Probably because he'd entrenched himself as the one "all-knowledgable" person in relation to all the systems that we had online. The company was probably afraid to be rid of him because they wouldn't have someone to maintain what they had.

I agree too that, if protection is requested and not approved fiscally, higher-ups should be held accountable for not protecting company assets.

I can honestly say...I've not had a virus scan program installed on my machine in *years*. I have a good firewall program and I have a good practice of *never* opening an email from someone I don't know...and I don't use the Microsoft Office suite because of inherent vulnerabilities in their architecture with respect to their interoperation.

Nonetheless...blatant incompetence and negligence should be grounds for dismissal...not malicious activities of techno ilk.

by dweeks

How about a bonus each year for those IT personnel that keep their system virus free. I know, I know, we're just doing our job.

by mattk

The problem is complex. First, IT personnel are hard to replace. Once you have been in a company a while, the IT infrastructure starts taking on a personality that is unique to the personnel. It is hard to find someone that can step in, step up, and get the job done. Second, Senior Management will not be fired. As a group they do not know how or why IT works. Poor decision making is hard to prove. Third, companies do not view IT as a business process. IT is an expense. If you asked a corporate president under what grounds he would fire his IT manager, he would probably not be able to answer the question. In reality, the answer is the same for all managers. Negligence, incompetance, and outright stupidity are hard to prove against an IT staffer... it is too easy to make an excuse for something going wrong.

Where I work, my manager hires newly trained people, who have tons of certificates and diploma's but less time in a server room than they have in school. I have more experience based on time alone than the rest of our four man department (been in IT since 1984), including the manager. I get paid less, do more, am called upon most often for difficult projects, and if something goes wrong, blamed first because I do not possess the paper. Our back up system has been failing for over three months. We have not had a good, complete back up for ninety days. I tried to talk to the manager about it and his response was that he is waiting for a new tape unit to be delivered before he takes action. I documented my concerns in an e-mail to him. I was advised that it is not a problem, and that I needed to remember who made the decisions in the department. I have a hard copy of that e-mail. We have already failed to provide restored files from back up on two occasions. I am waiting for the other shoe to drop as our drive arrays are having problems too. We are supposed to be a "minimum hard copy" company. Everything is on the network, and none is backed up. How can he get away with it? Nobody asks the question. If they do and they get a straight answer, he will tell them the new tape array is on the way. They will probably buy the story.

I have my hard copy.

by jsullo

What world are you living in? I have been in countless IT shops over the years and in many different positions and often everyone knows the vunerabilities but because of budgets, understaffing, vacations and all the other things that eat up time in the work place, i.e. new projects that are must haves. Which tend to come from non-technical people. These are all reasons stuff happens. I personally have been lucky not to have had any really major things happen but I do see how things that should be done to prevent get pushed aside in the course of normal business. So being this is the case who do you fire the CEO, CIO, Network Manager, Applications Manager, Technicians, who? You might say all, I would say get the company to rededicate themselves to protecting the network the way they should but then again if the money is not there then what is there to protect the files of a company in bankruptcy. You gotta have some real perspective on this. There are many things involved in failure so roll heads if you want, I'll concentrate on helping the company make money so we can use it to protect the systems.

Cheers

by pcpapa

I agree. People will make mistakes. The best solution feasible under the circumstances is not always the best solution. Where do you place the blame? Fire the techies for not doing the work? Fire middle management for not overseeing their operations better? Fire upper management for not allocating funds? And what makes you think the people that will replace them will be any better? Maybe you should just scrap the whole company and start over. The reality is that even if you do everything right you can still suffer from security issues, attacks, viruses, worms, etc. Sally Sue may still download a kiler virus and pass it around. Billy Bob may still find that website that introduces something undesirable to your network. You prepare the best plan possible and react in the best manner possible. You can't throw the baby out because the bath water is dirty and not bathing the baby isn't an option. :-)

by tony.savoie

Our WAN is made up of multiple domains. Each one with thier own set of admins and techs. My domain was the ONLY one not effected by the recent outbreak of Zotob, or any other outbreak I can remember.

Why? Because the techs and admins on our domain did thier job. Systems are always patched, virus patterns are up to date etc.

Zotob caused days of downtime at my company. All due to incompetence. Heads SHOULD be rolling, I dont understand why they're not.

by lanman235

Being the manager whos head would roll can only say this. Middle and upper management have no idea what the IT departments do. The have no idea what it takes to keep they systems running or what it takes to fix them when they're down. They only know we re there when they need us and invisible when they don't And that they have the toys they want when they want them.

by craig_b

Your question sounds like one that's loaded and is aimed at shaking things up, however I'll play along.
I think you have to look at the entire system and not just try to blame someone. Yes everyone should be held accountable. I could ask, why don't we take to justice the virus writters, or the software vendors that allow the security bugs, or Company Mgmt, or the IT dept, or the end users, or ...
The entire system needs to be worked on and improved. However Security is a double edge sword, simply disconnect your computer from the network permanently and you'll greatly cut down on security risks but that makes it harder to work with others. The point is their is always a cost vs risk factor and finding the right balance can be tricky. So I would recommend that you work on fixing the problems, instead of just trying to blame someone.

by tnc123

Yes indeed why on earth no one blames the computer guy. It must be strictly followed that if a virus or hacker attack happens in the company IT department must be held responsible.

I would think there must be some rules made and a code of cunduct must be formed relating to this

Jamal
travel-culture.com

by johanncox

When the heads use money or time as factors to deny obviously needed preventive measures, because "it has never happened to us" syndrome, they tend to quickly rally around each other.

They may also blame the IT crew unjustly. But, would not fire them, since the IT member(s) had already addressed the issue before it crippled.

Yes, DOCUMENT all conversations and suggestions!

When it comes to management cover each others back sides, it can be vigorous and zealous. When one starts to hit the slippery slope, others tend to follow.

When it IS the negligence of the IT member, sure, they should face the piper, but maybe not the hangman.

by robert.st.john

Larger corporations have the policy of NOT changing any thing until they have verified all the ramifications of what the patches COULD break, rather than look at what could happen by it NOT being patched. They can spend weeks months and years trying to evaluate a program or patch and there are places that have not even adopted XP service pack 2 yet, because it causes too many problems with the users.

by the admiral

I can give you one good reason. If we fired everyone once for what they do, then we would have a turnover that would make a standard helpdesk with a turnover of 400% look like the Miss America Pagent.

The fact of the matter is that, and this is a point that I drive home, if we are not going to enforce the policies that we are putting into place, then testing patches and virus updates and duplicating infastructure is a waste of time and money. Let alone, I think it is a waste of time and money to duplicate effort in the first place.

That is why heads do not roll. Eventually, you kick the dog enough and it will turn on you. You don't want employees telling executives to get rid of the IT department if you rely on them to ensure your systems are running.

by mvierling9

I think one of the major reasons that "heads not rolling" is because IT departments for the most part are largely understaffed. I work at a company of 55 employees and I'm the entire IT department (IT director, IT manager, help desk, network administrator, telecommuncation expert). When IT departments are continually being reduced, something has to give, and that is what we're seeing here.

by saintgeorge

Off with her head! Off with her head!

If you fire somebody on auto every time something goes wrong, you'll be soon working all alone, and you'll have to fire yourself because you will be the one resposible of creting a Gestapo office environment, of having everybody more interested in covering their own asses than doing a good job and, in the end, of depleting your workforce.

Even if the guys only resposibility was to keep up to date with every single security nuance that keeps popping up every day - and it is definitely not - mistakes are something to learn from. Of course, if somebody keeps making the same mistake over and over, then HE is making the mistake, not learning, and will have to be laid off so he will (possibly) learn something.

More often, punishment will come from other directions. No bonuses, no promotion, less confidence in that person. Things that will make him or her work to be better at his job.

In the end, management decisions do not follow a binary pattern, on or off, good or bad, do-everything-right or i'll-fire-your-sorry-ass. If you ever make management level, you'll learn that. Of course, it can always happen that you are so set in your ways that you won't learn ever and stay where you are.

Myself, I'd rather fire a low level employee who spends company time playing solitaire or second-guessing managerial decisions..

(Any similarity to actual people is purely coincidental. No offence is intended but you can inferr what you want.)

by montgomery gator

The original poster's suggestion would create a paranoid environment where people would be afraid to do anything proactive because they are afraid they will get fired. Similar to the Royal Navy in the 1700s when they hanged some Admirals to encourage others to do better.

by tewman2

Today in the tech world, there are always incompatibilities from one product to the next, when you fix one, another one doesn't work correctly, or vise versa.
Managers, although many don't understand technology, they do understand that the techs they have are the most familiar with the network they have. The techs know what works, or what it will take to change it to make something else compatible. If they fired a tech because of an issue as such, they would pay a lot more for someone else to come in and figure out what the last guy did to get it set up. Sure the new guy could tear it apart and rebuild it, but that would probably cost more valuable downtime.

I am a firm believer in using the latest patches, having everything to the latest patches. However, it takes time to test everything to make sure it?s going to work before pushing it out to the production floor. Sometimes you have to wait for a software vendor to patch their software before you can roll out the latest Microsoft patch. Sometimes the antivirus causes glitches with the fileserver committing files to disk. Use your heads think about it.

Did I mention that I am middle management

by awfernald

What I have observed that causes this type of problem is as follows:

1. Lack of coordination between technical areas (i.e. desktop support and network support) due to a variety of reasons including:
a) outsourcing of certain functions, especially desktop support;
b) political in-fighting between different managers
c) lack of documented savings for preventing the attacks vs. recovery from the attacks
d) lack of appropriate testing environments/procedures for performing the constant testing required to stay "up-to-date"
e) inexperienced technicians/administrators
f) lack of time to implement solution due to too much time fire-fighting

All of these can be overcome, but they require time, money or political intervention.

by rzimmerman

Cops don't get fired because they don't always catch the bad guys, especially if it is white collar crime like hacking. And Microsoft get's away scott free for creating these holes in the first place. Most IT techs are very conscientious but there are only so many hours in the day, and too many other things to do. Management needs to be held accountable first of all for ensuring that sufficient resources and tools are provided to prevent this problem. Also law enforcement needs to hire more cops with the training to catch these idiots.

by tewman2

In one situation I was in, upper management believed in me, they had seen me work miracles with their dilapidated equipment. However they didn?t want to release the funds to get the equipment that they really need for everything to function, as it should. The backup system failed, costing close to $10,000 in downtime, and reentry of data.
They knew it wasn?t my lack of ability that caused the problem. I had already proven my self. It was the equipment, this woke them up to reality, and opened up the budget.

by graphx

After being in a high profile magement job as an IT Director for a local meduim size multi-specialty Clinic with 50 doctors for several years, I can tell you that the issue is not that the IT Department doesn't want to put out the latest and greatest Anti-Virus, Gateway, Spyware software and tweak it to make it work. I mean really - how many geeks do you know who wouldn't want to get their hands on tweaking a new server, firewall, or internet appliance and make it purrr. It has to do with a more serious issue that these departments are connected to.

This entity being the Administration. I leterally had to beg for Anti-Virus for 2 years before Administration would cough up the funds to implement it. It took another year for spam and spyware appliances to be purchase because most of the time these guys only see the bottom line and the dollar signs attached to it.

This is actually more serious then most take it. Not only is it an issue when you go to purchase software or hardware, but also in aquiring new hands (staff) to implement the new products.

To much cutting corners and money saving ideas go in to effect by Administration, and since they are not versed in technology, I have seem most of the money saving methods for technology fail to save anymore money then just buying the product with no discounts or incentives and saving yourself downtime.

No matter how many times I personally addressed this with Administration, they dismissed my view points and decided to do it their way. Of course this just meant that I had more work and they wasted more of the doctors money then they saved.

As for the comment made about why don't the VP or CEO heads roll if they refuse the purchase. Are ya serious? Have you worked in Coorparate America? Fact is that they have reputations, money, a name for them selves - but actually doing the job that they know nothing about and you want them to manage to fix viruses problems and spyware problems when they do not allocate proper expense money or resouces to the cause. Come on.. think.

Thats whats up. Sometime the truth hurts!

by binarypc

One of the major problems with this statement is that virus writers are smarter than you are giving them credit for. They are writing the viruses to use necessary ports to Windows communication. A recent outbreak at a company we service caused the company to request port shut-downs throughout the environment. When they were shut down, GPO replication via FRS could no longer function. SMB connectivity between clients and servers across remote sites were shut down. Lots of little things were impacted.

In answer to your questions above, this large number of servers had virus patches and current OS patches. A full understanding of the viruses and what they are impacting is necessary before making snap judgements.

Of course, you could go ahead and fire your whole staff, then you would have noone to keep your environment healthy.

by beads

It all boils down to one of two things. One, IT was caught off guard or simply didn't heed the warnings. Or two, Management did not percieve active network management as being a big enough priority. Either way its no way to run your IT shop.

Firing everyone in the IT department over one incident would be a knee-jerk reaction. Now if this was an ongoing problem that happens more than has been recently reported it may be time to reasess your IT management in general.

My gut tells me its really a combination of the two above and isn't as black and white as it may appear at first glance.

- beads

by activated

This is the attitude that deserves being fired, why? Because accepting responsibility for the job you are hired to do is BASIC to having the job to begin with.

by bigmat

I don't know about your real world, but in mine there are all sorts of reasons the experts should not get fired.
The biggest reason is they are the experts....
Who are you going to get to do the job if you fired everyone that couldn't react fast enough because they had to fix someones paper jam or whatever?

The biggest reason experts do get fired is the ignorance of the people that are doing the firing.

How about the constraint of budget or the time it takes to fix an outbreak . A case in point SQL slammer took 15 minutes to spread to 90,000 computers, most companies virus defintion updates are not that fast to download or react, let alone the people writing the protection definition files.

How about the guru who has been struggling with finance department to buy a firewall device, only to find because finance management took 2 months to approve budget, the security of the hardware they put up in budget has been cracked by some virus scripting wizard and now needs more money to be spent to circumvent that risk, again another approval process.
How about we sacked all the experts, all you'd be left with are numb-skulls, then see how many viruses are introduced into your organisation.

Please remember we are only human so errors do occur, but on a percentage basis the statistic of error is very low.

I find it could be possible to read between the lines of "fcleroux's" thread one scenario I see is that fclleroux got fired for introducing a virus. Well I say if that is the case, don't come to me for a job because I just don't like your reasoning.

I recently saw a presentation where a expert told us about an email that was sent to staff from the Techs advising them not to open attachments if they didn't know who they were from. Half a day later the Techs sent out a bogus email from a bogus address within the organisation, 20 percent of the staff opened that attachment. The attachment reported back to the Techs as to who opened them. People are stupid, that's they way it is.

I probably have said too much, but please be honest with yourself.....

by activated

From a techie....If you dont defend the company as you should, you not only should be fired, you should be liable for damages as well.

by itguyy

When the car breaks do you fire the mechanic?

When the building catches on fire, do you repremand the firefighters?

When you have eight PC support guys covering 12 sites over the U.S. encompassing several thousand computers, continuely cut the budget, redirect them to new projects on a monthy basis, provide no training and do not listen to pleas for increased security, updates and allow departments to install their own operating systems on a regular basis as needed....

...Are you going to fire them for doing their best to keep up with virus writers who often have exploits out the day after the patches? Especially when several patches kill machines with non-standard installs?...

...Is the CIO (Presidents) going to fire the managers he has driving PC support who are simply working as hard as they can to keep up with the CIO's wishes and cost cuts?

..Is the CIO going to get fired for responding to the problem?

Not in this reality. Everyone works overtime, you get over it, and buy beer afterwards. Divide and conquer and try to get smarter next time.

-ITGuyy

by eddie15068

How do you know NOBODY has ever been fired? reads like you're in the know. I'd like to know where you work so I can be completely incompetent & keep my job. Sounds like you have an axe to grind. BTW how's SpyCatchers Alliance Inc.???

by eclark

Consider Cisco's IOS code leak on the web. Consider the fact that some of the leading anti-virus vendors release new definitions once per week. Consider that many malware applications do not detect all malware.

Are we to held accountable for the actions, or inaction of our vendors 100 percent of the time?

by d-joatmon

It's like asking "Why aren't all the police fired that don't catch every speeder on the freeway". Put in place of "police" your favorite IT scapegoat. Put in place of "speeder" every single threat to an IT environment. Got a new radar gun... guess what? The same company that built that radar gun is the same company that builds radar busters. Anybody ever wonder why MS goes into great detail in their "Hey we found another problem with the security of our OS"? Seems to me that it would be smarter to say, "Here's a patch we've fully tested, you need it, get it". As opposed to, "If you really want to screw up somebody's network here's a basic map of how to do it" each and every time they post a security patch.
It doesn't matter how many police are on the road there will always be speeders. Unless you're talking about an unfeasible 1 to 1 ratio it's a never ending problem. One IT guy to handle viruses. One IT guy to handle OS patches. One IT guy to handle firmware updates. The list goes on and on. Then if that 1 to 1 police doesn't catch his one speeder, then hell yes fire him.