Disaster Recovery

10 dumb things IT pros do that can mess up their networks

End users aren't the only ones whose misguided actions can bring a smooth-running network to a screeching halt. IT pros make their share of mistakes, too -- from sliding on DR planning to stalling on repairs to ignoring the need for logs and documentation.

End users aren't the only ones whose misguided actions can bring a smooth-running network to a screeching halt. IT pros make their share of mistakes, too -- from sliding on DR planning to stalling on repairs to ignoring the need for logs and documentation.


One of the most popular pastimes of IT professionals is complaining about the dumb things users do. We all get a laugh from articles like TechRepublic's ultimate collection of dumb user stories. But if we're honest, we have to admit that computer novices aren't the only ones who make mistakes. Most network administrators could (but probably won't) tell you about their "most embarrassing moment." That's the one where you discover you accidentally misconfigured the firewall to shut down the boss's Internet connection or that the backup you've been making every day has been copying the wrong files. Oops.

Let's take a look at some of the most common dumb things IT pros do that can mess up their networks -- and how you can avoid making such mistakes yourself.

Note: This information also appears in article format and is available as a PDF download.

#1: Don't have a comprehensive backup and disaster recovery plan

It's not that backing up is hard to do. The problem is that it sometimes gets lost in the shuffle, because most network administrators are overloaded already, and backups are something that seem like a waste of time and effort--until you need them.

Of course you back up your organization's important data. I'm not suggesting that most admins don't have a backup strategy in place. But many of those backup strategies haven't changed in decades. You set up a tape backup to copy certain important files at specified intervals and then forget about it. You don't get around to assessing and updating that backup strategy -- or even testing the tapes periodically to make sure your data really is getting backed up -- until something forces you to do so (the tape system breaks or worse, you have a catastrophic data loss that forces you to actually use those backups).

It's even worse when it comes to full-fledged disaster recovery plans. You may have a written business continuity plan languishing in a drawer somewhere, but is it really up to date? Does it take into account all of your current equipment and personnel? Are all critical personnel aware of the plan? (For instance, new people may have been hired into key positions since the time the plan was formulated.) Does the plan cover all important elements, including how to detect the problem as quickly as possible, how to notify affected persons, how to isolate affected systems, and what actions to take to repair the damage and restore productivity?

#2: Ignore warning signs

That UPS has been showing signs of giving up the ghost for weeks. Or the mail server is suddenly having to be rebooted several times per day. Users are complaining that their Web connectivity mysteriously drops for a few minutes and then comes back. But things are still working, sort of, so you put off investigating the problem until the day you come into work and network is down.

As with our physical health, it pays to heed early warning signs that something is wrong with the network and catch it before it becomes more serious.

#3: Never document changes

When you make changes to the server's configuration settings, it pays to take the time to document them. You'll be glad you did if a physical disaster destroys the machine or the operating system fails and you have to start over from scratch. Circumstances don't even have to be that drastic; what if you just make new changes that don't work the way you expected, and you don't quite remember the old settings?

Sure, it takes a little time, but like backing up, it's worth the effort.

#4: Don't waste space on logging

One way to save hard disk space is to forego enabling logging or set your log files to overwrite at a small file size threshold. The problem with that is that disk space is relatively cheap, but hours of pulling your hair out when you're trying to troubleshoot a problem without logs to help you discover what happened can be costly, in terms of both money and frustration.

Some applications don't have their logs turned on automatically. But if you want to save yourself a lot of grief when something goes wrong, adopt the philosophy of "everything that can be logged should be logged."

#5: Take your time about installing critical updates

The "It'll never happen to me" syndrome has been the downfall of many networks. Yes, updates and patches sometimes break important applications, cause connectivity problems, or even crash the operating system. You should thoroughly test upgrades before you roll them out to prevent such occurrences. But you should do so as quickly as possible and get those updates installed once you've determined that they're safe.

Many major virus or worm infestations have done untold damage to systems even though the patches for them had already been released.

#6: Save time and money by putting off upgrades

Upgrading your operating systems and mission-critical applications can be time consuming and expensive. But putting off upgrades for too long can cost you even more, especially in terms of security. There are a couple of reasons for that:

  • New software usually has more security mechanisms built in. There is a much greater focus on writing secure code today than in years past.
  • Vendors generally retire support for older software after awhile. That means they stop releasing security patches for it, so if you're running the old stuff, you may not be protected against new vulnerabilities.

If upgrading all the systems in your organization isn't feasible, do the upgrade in stages, concentrating on the most exposed systems first.

#7: Manage passwords sloppily

Although multifactor authentication (smart cards, biometrics) is becoming more popular, most organizations still depend on user names and passwords to log onto the network. Bad password policies and sloppy password management create a weak link that can allow attackers to invade your systems with little technical skill needed.

Require lengthy, complex passwords (or better, passphrases), require users to change them frequently, and don't allow reuse of the same passwords over and over. Enforce password policies through Windows group policy or third-party products. Ensure that users are educated about the necessity to keep passwords confidential and are forewarned about the techniques that social engineers may use to discover their passwords.

If at all possible, implement a second authentication method (something you have or something you are) in addition to the password or PIN (something you know).

#8: Try to please all the people all of the time

Network administration isn't the job for someone who needs to be liked by everyone. You'll often be setting down and enforcing rules that users don't like. Resist the temptation to make exceptions ("Okay, we'll configure the firewall to allow you to use instant messaging since you asked so nicely.")

It's your job to see that users have the access they need to do their jobs -- and no more.

#9: Don't try to please any of the people any of the time

Just as it's important to stand your ground when the security or integrity of the network is at stake, it's also important to listen to both management and your users, find out what they do need to do their jobs, and make it as easy for them as you can--within the parameters of your mission (a secure and reliable network).

Don't lose sight of the reason the network exists in the first place: so that users can share files and devices, send and receive mail, access the Internet, etc. If you make those tasks unnecessarily difficult for them, they'll just look for ways to circumvent your security measures, possibly introducing even worse threats.

#10: Make yourself indispensable by not training anyone else to do your job

This is a common mistake throughout the business world, not just in IT. You think if you're the only one who knows how the mail server is configured or where all the switches are, your job will be secure. This is another reason some administrators fail to document the network configuration and changes.

The sad fact is: no one is indispensable. If you got hit by a truck tomorrow, the company would go on. Your secrecy might make things a lot more difficult for your successor, but eventually he or she will figure it out.

In the meantime, by failing to train others to do your tasks, you may lock yourself into a position that makes it harder to get a promotion... or even take a vacation.

About

Debra Littlejohn Shinder, MCSE, MVP is a technology consultant, trainer, and writer who has authored a number of books on computer operating systems, networking, and security. Deb is a tech editor, developmental editor, and contributor to over 20 add...

29 comments
kenboone
kenboone

Our network was shut down by a rogue Apple Airport that the user selected to be a DNS server. It could give out IP addresses faster than our DNS servers.

calberty
calberty

I myself always backup the system state file and the registry anytime I make a change on any of my servers. I have recovered several servers but doing this.

The 'G-Man.'
The 'G-Man.'

Is totally off topic and has nothing to do with messing up networks but a career tip. It looks to have been added just to get some conversation going.

Doug Vitale
Doug Vitale

To achieve the most efficient network administration and troubleshooting, you should label cables and all the components in your server racks so that it's immediately clear what you're looking at. Otherwise you end up wasting a lot of time. You should also keep server rooms/comm closets free of clutter.

dallen
dallen

The information is certainly important but the layout needs some attentiom. On my IE7 browser half the headings overlap the top line of the following paragraph making it hard to read. Why should I have to download the PDF to actually be able to read the article?

britnat
britnat

Let's add number 11: "don't fix it if it isn't broken." Some administrators seem to love tinkering. (And also don't keep notes). Every now and again it goes massively wrong. In some cases this leads to a "panic rebuild" of the system, all completely undocumented, and with minimal security.

Marty R. Milette
Marty R. Milette

IMHO, there is no point in logging what nobody will ever look at. For example, unless you work at Microsoft, the server crash dumps are virtually useless. Audit logs (on by default) can consume not only disk space but processor resources. Unless looking for a specific issue -- normally turn success audits off. IIS logs can take huge amounts of space and processor resources -- do you really need them? (Especially if you are using a different statistics system that doesn't use them.) Processor resources are more of a consideration than disk space with many forms of logging.

LarryD4
LarryD4

In my opinion these items are part of the IT Pro Job role. But 90% of the time these issues occur due to lack of management. It is the job of the IT Manager/Supervisor to assign reponsiblities broadly across mutiple employees acting as backups to others. Issues mentioned in the article occur due to lack of management. Granted any one person could not document and/or fail to monitor and upgrade. But its the person in charge who should be monitoring these responsibilities. So perhaps the next blog title should be.. The 10 dumb things IT Managers fail to monitor...

tech10171968
tech10171968

I've found this to be all too true in my personal experience. In our admittedly small company I'm the only one who knows anything about IT at all; everyone else here has neither the requisite computer literacy nor the training to fill in as a temporary sysadmin while I'm away. Naturally this means phone calls while I'm sick or on vacation and, sometimes, I can't even run a simple errand during the day without getting an IT-related call or text message. Fortunately we recently hired a new technician who was also a comp. sci. major, and I've been training him every chance I get. Hopefully he can take some of the load off my back when I have to leave the office for any reason.

icbhod
icbhod

#10: Make yourself indispensable by not training anyone else to do your job I got burned for years at the last company I was with because my boss felt like nobody could do my job....."he who cannot be replaced cannot be promted"

JSully
JSully

Very good article and a reminder of many of the mundane, boring tasks that any good administrator must do on a regular basis to keep the network up. I do believe though that the onus falls on Manegement to identify "who does what" and "who owns what" to clearly define expecations with Adminsand hold them accountable. I REALLY liked #9 and the talk of users cirumventing security measures because of excessive and inflexible IT controls. The network is provided for the Business to get their jobs done!

larrie_jr
larrie_jr

My IE 7 browser worked this and all tech republic articles correctly. I didn't download it, but went to the blog section if that helps at all... but I would look in my settings for an issue, not tech republic

larrie_jr
larrie_jr

The relativly small files necessary for these functions require minimal procesor resources and the file sizes may be limited to avoid data bleed. Think of this as insurance... you may not always need it, but it is a life saver when you do... the question now becomes not whether to log or not to log, but what to log. Yes, the server crash dumps are pretty useless in and of themselves, but what about when it is cross refrenced with another log that would clarify to you WHY or HOW it happened CIO/CTO perhaps you have a bit of the 'budget frights' and have lost the perspective of ROI/TCO vs QoS

shryko
shryko

As someone in accounting, I can assure you, if the IRS ever ***MIGHT*** come through your door? they will want *EVERY* piece of information. whether you think it matters or not... they will decide what is relevant. if you can't provide it, because you didn't keep it? heaven help you, cause the fines can pile up quickly, and they *CAN* add interest, dating the fine to when you stopped keeping the information... The Canadian equivalent to the IRS can audit up to 10 years back... that's 10 years of information to keep on hand... you don't need everything, but running the risk just isn't worth it. (oh, and the IRS can toss you behind bars... another added benefit to the mountain of fines they can bury you under.)

rblanc
rblanc

I think that not logging is a call for problems in the future. There are many reasons for this, but to mention a few: - investigation of internal threat: costs of investigating an internal attack (be it intentional or accidental) grows immensely if your network/organization is just a tad complex. Trying to figure out what happened, when, and how if you don't have any proof is a problem that can bring huge headaches (and sleepless nights). - deterrence of internal threat: knowing that all of your actions (specially privileged users') will be logged and possible audited will at least stop a part of the possible attacks. - internal auditing processess: how to conduct this if you don't have the necessary information? - forensic readiness: the average cost of one sinlge e-discovery process went up to $250.000, with most of it going for the reviewing of electronic records. And about 70% of the documents reviewed proved to be irrelevant for the investigated case. So... why not building up all the possible evidence BEFORE you may actually need it in case of litigation? And there's more... in the end, logs need to be seen as an insurance. It's something you may "pay" all your life and never need it... but when you do need it, phew, thank goodness it was there. I would have to add to the logging issue, though, something that I think it's of the outmost importance: logs have to be protected (ideally tamper-evident). All these argumentation for logging completetly dissapear if there's any kind of manipulation in them. Even worse, if manipulated they could be falsely accepted as truthful evidence. I work at a company that focuse precisely on this issue (http://www.kinamik.com).

Chaz Chance#
Chaz Chance#

I am tied to my present position because of lack of a replacement. Over the years I have trained 8 different people to do the part of my role where I am seen as "irreplaceable". Each has moved on as quickly as they could. The trouble is that it is something that a trained monkey could do, and many people would happily do for half my salary. But the company I work for insists on hiring people with degrees, rather than the right people for the job. So I get financial pats on the head, and my career stalls.

straightp
straightp

you just have to hope your boss doesnt figure it out.

steve-f
steve-f

Cool article. BTW the link to the dumb user stories is broken, and I would really love to read this!

Marty R. Milette
Marty R. Milette

I have to disagree with this statement for two reasons: "The relativly small files necessary for these functions require minimal procesor resources and the file sizes may be limited to avoid data bleed." First, you obviously have no idea just how much data and processor time can be consumed through excess logging. Go see your friendly neighbourhood system engineer and have him (or her) demonstrate on a busy server. And while you're there -- ask him how long it would take to extract useful information from that mass of data without using some very expensive third-party tools. Second, yes, you can limit the size of the files for SOME logging operations (Windows standard server logs) -- other applications may NOT give you that flexibility. Additionally, by limiting the size of the files, you're defeating the purpose. Some security incidents can generate millions of entries in a few minutes -- filling up the log and wrapping around to destroy the first entry. Alternatively, leaving logging unlimited can fill the disk and potentially crash the server. Much of my work involves specifying COST-EFFECTIVE, HIGH-PERFORMANCE hardware and software configurations to meet business requirements. I also need to account for human resources and decide where their time is best spent. I'd suspect that anyone who does NOT have 'budget frights' either works in the public service -- where money is no object -- or has a much better technique for squeezing money out of management than I do. Can you protect against every possible contingency? No, of course not. Everything in this business involves trade-offs. If you want 100% fully-redundant systems, that means spending more than twice as much -- so we use INTELLIGENCE to assess each risk and decide where we should spend our limited budgets. Again, at no point did I say not to log or audit -- but you need to use your brain and decide exactly WHY you are doing it, and WHAT problems you expect to solve by doing it, and exactly how much resources you can affort to spend doing it -- in hardware, software and human resources. Unfortunately, this is a fairly complex topic that can't be described properly in any way with a simple bullet point that says logging=good, not logging=evil. Logging and audit is only a portion of a proper risk assessment / mitigation plan -- which, depending on the environment may take anywhere from a day to several months to do.

Marty R. Milette
Marty R. Milette

Do you know what type of auditing and logging we are talking about in this post? Also, when did Revenue Canada change financial records retention from 7 years to 10?

Meesha
Meesha

First hand experience with this issue. Over the 2007/08 holiday season, the offices were shut down. Yet one of our front facing (internet) systems which normally has never given any problems went down. The tech decided not to answer the alert page believing that it was due to an automated back up process. Well it wasn't and before anyone knew the system had been down for over a week. When I asked for the logs to review what happened, apparently this same tech decided that the logs were just jamming up the server so did not turn them on. Needless to say, that tech is no longer that tech.

Marty R. Milette
Marty R. Milette

I didn't say that NO auditing and logging was to be implement, but that it should be done INTELLIGENTLY and APPROPRIATELY. Just enabling every possible form of auditing and logging is simply stupid. It will collect a million times more data than will ever be necessary and consume valuable resources that are better used elsewhere. Take the simple case of enabling successful access audits on a single network drive resource. Sure, you'll know who was accessing every file on the drive and when -- but that is an incredibly expensive price to pay for the terrabytes of data and processor resources you'll lose just to log it -- let along the costs to archive and store it. "Dumb - capture it all and let god sort it out" logging and audit is NOT the answer. God doesn't work in any of my customer's data centers -- and they aren't about to hire extra staff and hardware to deal with that volume of 99.99999% worthless data. Don't believe it? Enable all auditing on one of your busy network drives and see how long it takes the logs to fill the entire remaining capacity of the disk. And while you're at it -- monitor the processor resources consumed doing it. System administrators need to use laser precision in determining what to audit and when. Unfortunately, this requires a considerable amount more knowledge and skill than the 'dumb' approach. It also means that someone has to have it in their job description to actually go in and analyze the data collected -- no point in doing it otherwise. (Better have some good and very expensive tools to help with that task as well.)

GreenPirogue
GreenPirogue

You wrote: "We are all expendable. You just have to hope your boss doesnt figure it out." Well, the boss knows he is expendable as well. But when you add value to a company, why on earth would a competent boss fire you?

rblanc
rblanc

...as you probably know, the decision on what to log, what to keep and for how long is not a decision that is on IT ground. It's a cross-functional decision that should include the legal, IT, operations and security departments.

rblanc
rblanc

I understand you didn't mean DO NOT log, just I didn't mean LOG EVERYTHING. It is obvious that there may be some logs that you may think are irrelevant, and they may produce tons of "useless" data. Knowing who accessed what file on a specific network may not important now (and it may never be important in the future either), but if you find yourself on a litigation hold, with the need to produce admissible evidence before a specific deadline, I am positive that you (or me) would consider that piece of information of tremendous value. As somebody mentioned before, these kind of logs will provide you an important piece of information for getting a clear and broader idea of whatever happened. It's an insurance policy. These logs are part of the big picture, not the picture itself.

Kevin.Dorrell
Kevin.Dorrell

I worked for a company once that had a simple policy on this. If anyone was indispensible, they would be fired.

eric
eric

I think that you can end up looking more expendable if you don't take the basic steps to create redundancy for yourself. So in turn by creating that redundancy it makes you more unexpendable. (in the eyes of a competent boss).