After Hours·5,428 discussions

What’s the worst mistake you ever made as an IT pro?

Locked

We talk a lot (and laugh a lot) about the dumb things our users do–but of course, we do plenty of dumb things ourselves.

The recent article (http://techrepublic.com.com/5100-10877-6044807.html) and download (http://techrepublic.com.com/5138-10877-6040785.html) “10 classic clueless-user stories” showcases bizarre tales of user behavior as described by some TechRepublic members. Now let’s turn the tables and talk about OUR mistakes.

What’s the biggest blunder you ever made (or almost made) during your IT career? Don’t hold back! Share it here or pop over to my blog and see what people are saying.

Jody

Topic

Not a technical blunder

In reply to What’s the worst mistake you ever made as an IT pro?

The worst mistake I’ve made in my career is one that I’ve made several times: trusting people who turn out not to be trustworthy. I’ve lost a couple of jobs by being the fall guy. I’ve made some decisions based upon what I was told by people I trusted which ruined me. I’m still paying for them. I should be retired by now but I’m still working because of misplaced trust.

Still, I can’t get over the fact that not ever trusting anyone has got to be an even more dismal life.

We’ve all made errors

In reply to Not a technical blunder

but I suppose my big time “gaff” was asking someone at work to delete files X,Y,and Z off one of the desktops,whilst I went to lunch.Dont worry,I said,they’re not important,as I’ve finished those “work-tickets”,so there’s no point keeping them on file.One hour later,I returned to find the “said person” had misheard me,and was formating the entire hard drive.Forunately,everything had been backed-up elsewhere,as disembowlment would surelly have followed,not to mention my P45.My boss however,was still not amused.Guess what I spent doing all afternoon?

Reply To: What’s the worst mistake you ever made as an IT pro?

In reply to We’ve all made errors

Writing Service pack 3 to disguise a disk full of porn

Going into IT believing pay was good

In reply to Reply To: What’s the worst mistake you ever made as an IT pro?

I should have gone into Sales…. The salaries are better than most but hell I could drive a union garbage truck to get the same pay with no worry of being outsourced.

Disabled NetBIOS on a workstation…

In reply to Reply To: What’s the worst mistake you ever made as an IT pro?

because of the usual funky security issues with NetBIOS being enabled on a workstation. It was a workstation I used all the time and then all of a sudden, couldn’t get Net access anymore. It bugged me for the longest about it. Tried figuring out for months why. Then, some months ago (the problem started during the summer of last year) the network admin was doing some work to the workstations on the same row with that workstation, so I told her the problem. Turns out the problem was that the disabled NetBIOS setting was the problem. The DNS server has NetBIOS enabled, so of course the workstations needs the setting info from the server for Net access.

Dum-dum me should’ve known that anyway. Another tale of trying to be clever when you really aren’t being clever *lol*.

This sounds like a user error

In reply to We’ve all made errors

This wasn’t your fault, but another stupid user fault.

Newbie troubles

In reply to We’ve all made errors

In my IT beggining I was reinstalling our finantial application (MSDOS back on 1988) and I thought I’ve made a backup of the data directory and I’ve erased it. Thinking better (but too late) I’ve managed to find the place where the backup was and… It wasn’t.
5 years of Company’s Finantial just vanished under my hands (our backup policy sucked), I was already feeling myself unemployed.
Then I called (in secret) our consultor and he introduced me to Norton Utilities and we’ve recovered those %&$#&%$ files one by one. In those days I thought he was some kind of miracle maker.

That feeling…

In reply to Newbie troubles

was one expressed to me by more than one user when I worked as an applications expert. Win 95 was the rage then, but a few of our customers still used DOS-based software and, occasionally, ran into problems. I still remembered enough (simple) DOS commands to get them out of their holes– and their admiration at my “miracle working” shone through the phone.

Same here

In reply to Newbie troubles

This was years ago.. around 1998 i think. I once had someones PC here was formatting the drive in my computer.. well at least i thought I was. I have removable hard drive racks and there are 2 hard drives in the PC, one in each rack. I took one (the C: drive) out which should have been the first drive when rebooting from a boot disk, after all it was the master on the first ribbon, but for some unknown reason after booting off of a boot disk the 2nd drive, which was on the 2nd ribbon suddenly became the C: drive and the c: drive became the d: drive.

I never looked to check that the c: drive was the first drive and typed in to format the C: drive. after I took it out and put my regular drive back in I found that my 2nd drive, which was used for storing files and stuff, had been formatted.. Needless to say I could have kicked myself in the butt for doing it and not double checking.. I never did that again..

writing /dev/hda

In reply to We’ve all made errors

what about this:
grep “something” > /dev/hda1

wasted windows

In reply to Not a technical blunder

in my first reall job, hired straight out of internship, i decided that i neither had to back up the boss’s laptop, nor actually follow the instructions for putting in the new pcmcia nic. i simply popped it in, ran the install, and it all went to hell.
This was on Win95, by the way.

ended up having to wipe the device from scratch because so many thing failed after that. this was before i knew about imaging, before i knew just about anything really. i had only ever worked on my own, single machine at home. I was THIS close to getting fired.

AD account passwords

In reply to wasted windows

When I started at my current job, our Active Directory was mainly used to for access to an intranet site and none of the client desktops were Windows 2000 or on the domain yet. I downloaded a software package of “NT Tools” and accidentally set everyone’s password to expire. Of course with no way for users to reset their own password, i spent the next three days managing “special” voice mail box setup specifical for people who need to login, calling them back individually to give them a new password. Great way to filter out unused accounts………….

Took down entire network with Cisco VTP Domain blunder

In reply to AD account passwords

I didn’t think much of what I was doing at the time. I was adding a Cisco Catalyst switch to two other switches in a high-availablity data center. This new switch was recycled from a call center and was already configured with its own VLAN database… and as a VTP Domain Server! I didn’t clear the config on it, so when I connected it to one of the other switches, and set that connection as a trunk, the VTP Domain Server on the new switch contended with the VTP Domain Server on one of the other switches (you can only have one!). Since the switch I added had a higher rev. code, it won, and proceeded to erase all of the existing VLAN information on the other two switches, rendering every single port disabled. It was also the middle of a regular business day, so the effects were immediately felt.

It took me about 3 minutes to rectify this blunder once I was alerted. To my surprise I didn’t get fired, but the director forwarded me an email from the CEO that read, “WHAT THE BLOODY HELL HAPPENED?” with his note added saying, “I don’t ever want to receive emails like this again.”

The infemous…

In reply to Took down entire network with Cisco VTP Domain blunder

debug all on the central 7206 router that had no redundancy. It happen at around 14h

Along the same lines

In reply to AD account passwords

In the NT domain days, I ran an upgrade/patch CD provided by a maintenance contractor without testing first. I was under the management gun because business critical applications were failing without the patches provided. Long story short, the patches provided “fixed” their products but broke others…and went into the NT accounts and set them all to “Never Expire” which was a violation of regulatory requirements for my industry. I spent the rest of the week helping to fix all the applications that broke a touching each account to uncheck the expiration box.

clear ip bgp * OOPS!

In reply to Along the same lines

Not me but a colleague who was an employee of a large ISP with a 3 letter title. Got back from a Cisco routing course and thought they would try out some commands on the core network. It took 45 minutes for the core to reconverge. P45 followed shortly afterwards.

Hard Drive Replacement

In reply to AD account passwords

Back when I was new at using GHOST I had a client who wanted to upgrade a 20 GB HDD to a 40 GB HDD. Long story short I ended up GHOSTing the blank 40 GB drive over the full 20 GB drive. Thus I began living by the motto: “Before you move forward make sure you can go backwards.”
The client I had was pretty cool about the whole thing. He had actually backup but his Quickbook databases before I hammered his drive. The reset of the data was stuff he said he could live without.

I didn’t think anybody else…

In reply to Hard Drive Replacement

was that dumb. When I did it (to my own !@#$ machine), it gave me the basis of a new personal rule: No OS (re-)installs after 10 pm.

Killed by a power supply

In reply to wasted windows

Back in the days when we were using systems with the old AT power supplies, I had to replace one that was about three months old on a system owned by the administrator of a Medical center. It had about 350 meg of data, (DOS) none of which was backed up. After I plugged in the new power supply I turned the system on. My first thought was where the siren sound was coming from. When I realized it was the CPU fan running at about 10,000 RPM, it was too late. Various caps on the system board started exploding. In short order the room was filled full of “magic smoke” (All electronics depend on magic smoke to operate. If you let the magic smoke out, the said electronic device no longer works) All this happened before I could get the switch on the power supply turned off. Upon removing the power supply and testing it with a volt meter, I found the 12 volt circuits really had around 36 volts and the 5 volt lines had were in the 18 volt range. We obviously had a new. defective power supply. Unfortunately it took pretty much every component out of that system, including the hard drive. The customer was livid, the fact that I this was the first and only time I have ever had this happen made no difference to him.
From then on, we tested every replacement power supply when it was unpacked.

Reply To: What’s the worst mistake you ever made as an IT pro?

In reply to Killed by a power supply

We here in Oz get power supplies sometimes set to 110Volts. They can make a pretty display!
I was installing such a beast (no external switch to set input voltage: it was by internal link), in a Unix server some years ago, in front of an anxious customer. Yes, it went bang (very loud!). His calm and collected response? “It’s not supposed to do that, is it?”

Reply To: What’s the worst mistake you ever made as an IT pro?

In reply to Reply To: What’s the worst mistake you ever made as an IT pro?

I can see where that could be a real problem. I have had a number of occasions where the switch gets flipped to 220, here absolutely nothing happens. I had a friend whose home system died. His wife’s friend worked on it and declared the power supply to be dead. They asked me to get them a new one, which I did. When I started to take the old one out, I noticed the switch was set on 220. I flipped it back and saved him the cost of the new power supply. They think I am a genius. I will not dispel that notion.

My First Power Supply

In reply to Reply To: What’s the worst mistake you ever made as an IT pro?

I had never changed power supplies before and ‘inadvertently’ got the wires mixed up. When I powered up the system it took out a fuse in a breaker box covering half the building. This impacted many computers including the mini running 95% of the company. Fortunately it had an internal battery and surge protector so it powered down gracefully. The Operations Manager and my boss the Systems Analyst got very excited but fortunately never figured what caused the ‘mysteries power surge’, nor did I clue them in! I now properly label all wires as they become disconnected, no matter how simple the circuit!

somewhat similiar problem with friend

In reply to Reply To: What’s the worst mistake you ever made as an IT pro?

I had a friend that something like that had happened to. She knew someone that was suppost to be real good with computers and he needed a little extra money and she was having problems with her computer so she paid him to fix it. He lloked at it and told her that the hard drive was bad and she needed a new one. He even took the old one home and tried it on his computer to make sure it was bad. he still said that the drive was bad.

after this she talked to me and I asked to see the drive. Reconfigured the jumpers to the right settings and wouldnt you know it, it started right up.. I asked her what she wanted off of it. She was shocked that I got it to work, after all, the guy that she hired tried this drive in 2 computers and could not get it to work. I told her to bring her computer over and I checked it at my house. First thing I noticed was that the CDROM was not working, nor was it getting any power. Found out that the lines that were coming out of the power supply that led to the CDROM and the hard drive was glitching and not giving power at times.. replaced the power supply and she has not had a problem sense..

Needless to say that she never had him look at her computer again.

+5 and +12 reversed on a Dell server

In reply to Killed by a power supply

Same situation except the connector was wired backwards (5 got 12 and 12 got 5). Everything on the SCSI bus was fried, including the hard drives! Everyone at Dell knew my name that day.

intern job

In reply to Killed by a power supply

when I was an intern from school they sent me over to help out at an organization. when I got there things were really messed up, they had computers laying all around, most of them had things wrong with them. I found somehad fraid and/or missing wires, missing ram, passwords on CMOS, and more.

I got to one computer that did not work. I was told that the previous intern had worked on it, but it had not worked sinse then. I took a look at it and found that he for some unknown reason, took the power plug off of the motherboard and then put it back on. The problem was that the computer was one of the old AT styles and he put did put the power plugs back on, but backwards.. needless to say that it never ram again.

Forgot to backup user’s files

In reply to wasted windows

This was one of those days I was going to call in sick, but decided to be a ‘trooper’ and come in anyways…

Part of the image process is to run a backup script, which I just forgot to run and blew the system away.
There was no backups or anything.
Lucky for me this user wasn’t a big fish or else I’d be canned for sure. The worst is that feeling in the back of your throat when you’re about to tell the user that all his files from the last 2 years are gone….

Thats a mistake made by most of us.

In reply to Not a technical blunder

My biggest mistake was trusting non-technical people, who were brought in to handle sales and business managment. They are retired and rich now, and I’m still working for a living.

Electricity

In reply to What’s the worst mistake you ever made as an IT pro?

I insisted that a PC I was working on had a bad power supply. I filled out the work order (warranty), tried fixing it myself, called for advice from a co-worker, looked on the back for the serial number, and called for repair. When I was moving it to put it on the repair shelf, I realized that I’d made a terrible mistake. It wasn’t plugged in. Imagine, they need electricity to run.

Wouldn’t be so bad if I could have kept it quiet. No, I had to call and cancel the work order, and they wouldn’t take “I fixed the problem” as an excuse. I should’ve just opened the case, and ripped out some wires and turned it in for repair.

DB cleansing

In reply to Electricity

Did you ever had the need to delete some old movements records from a couple of table in a live (worldwide running TP) DB ?
Double check and have someone looking from your back to avoid to mistype “where year=2005” with “where year=2006” !!!

After a couple of hours, a terrible belly ache and thousand of angry phone call, I recovered almost everything from the night backup. Whew!!

IT Support Rule #1

In reply to Electricity

IT rule #1 was obviously ignored–always check the power plug before you call for repair.

Can I relate to the power plug issue?

In reply to IT Support Rule #1

Back when I was a newbie, I couldn’t figure out why the computer would not work. I checked all the plug-ins that were connected directly to the computer, pressed all the right buttons but neglected to check the other end of the power supply cord. Duh!

Ex-CompUSA Employee?

In reply to Electricity

That is a repeat of what I have heard many times from those haha…er..*savvy* A+ Techs that work in a CompUSA repair dept. They are computer reapir men by training, not by experience 🙂 but I won’t say I’ve never felt like kicking a computer off the roof after 20 minutes of hair tearing mayhem, and YES it was plug in. I just forgot the…uhh…err…the little switch on the back of the power supply!!

Giddy

In reply to Ex-CompUSA Employee?

Nope, I was just so giddy at having quit the job from hell, that all brains flew out of my head. It was my last week in a job where there was one IT person whose sole job was to read everyone’s email, and forward the “seditious” ones to management.

i know the feeling

In reply to Ex-CompUSA Employee?

I once had a computer that i was checking out.. I had the thing started and then had to move it for some reason. I tried turning it back on but it would not boot up. For the life of me, I could not figure it out. I even went as far as checking to make sure the cables were pluged in, whcih they were, and even took the side of the case off and checked all the cables inside to see if one came loose. Imagine how stupid I felt after messing with this thing for a good half hour just to find that I bumped that little flip switch on the power supply and killed all power to the PC.

Electrical related

In reply to Electricity

I was covering LAN admin as well as telcom duties at a high tech electronics Mfr. I went into a telcom room to change a phone connection and leaned back against a breaker subpanel. I took out the whole engineering department. You just can’t turn a breaker back on fast enough. thankfully there was no serious fallout or loss of data (or job).

A good excuse

In reply to Electricity

See, you have to be able to think of a good excuse on your feet.

I once did a similar thing I asked another tech if we had any spare power supplies after my [faulty] diagnosis. He went to get one, and I noticed the thing wasn’t plugged in. I plugged it in and tried the PC with power. It powered up just fine. Just then the tech came back with the PSU. Thinking quickly, I grabbed a spare power cord that was lying on my bench and threw it away. “Nevermind,” I said, “it was a bad power cable.” After he left for the day I retrieved the discarded “bad” cable.

More electricity fun

In reply to Electricity

This one wasn’t my fault, but back in the PDP11 days I was installing a system at a electrical contracting company. They did their own wiring. I always checked the outlet voltage before I plugged a system in. I had just told my co-worker not to plug in the system until I got back with a meter. I had just walked out of the room when I heard a sound like a gun going off followed by the lights blinking. Turns out the electrician wired the 110 outlet for 220. Not only did it trip the breaker but it destroyed the surge suppressors in the rack power distribution unit. It is a good thing I had a habit of disconnecting the system components from the PDU. Needless to say the owner of the company was a little upset about the $500 bill for a new PDU. Don’t know what happened to the electrician. But, I kept my eye on my coworker after that.

My biggest oops was when I was fixing a disk drive on a Cray. Put in a new power supply and forgot to check the voltage switch. It was set to 220 instead of 110. Back in those days the power supplies were not auto sensing. Fired it up and snap say goodbye to a $2000 power supply. The good thing was that my district manager came from a field service background. His response was “Well when you work in the field long enough you are bound to blow something up.” Have to love it when you have an understanding boss.

No equipment harmed, just me…

In reply to Electricity

I once decided that the cause of some grimy printouts from a laser printer was the accumulation of toner inside the printer. So I decided to vacuum it out, forgetting the cardinal rule of unplugging equipment before working on it. The vacuum cleaner had a metal hose, and I must have touched the corona wire. After doing a little dance and picking myself off the floor, I looked around to make sure no-one had seen me and vowed to always turn stuff off in future. I am still interested to find out exactly how much electricity went through me that day, but it felt like a hell of a lot!

As a student,

In reply to What’s the worst mistake you ever made as an IT pro?

Which wasn’t that long ago for me. I made the mistake of switching the little red switch on the back of most PSU’s from 240v to 110v. Not only did I fry the power supply but it also tripped the RCD taking out power to the entire wing.

Fortunatly no PC’s were harmed in the making of that mistake.

Mike

As a newbie at Tech Company i once worked for

In reply to As a student,

I deleted the CEO’s mail box after recieving the wrong description of the account that was supposed to be removed from the network.Luckily managed to recover all the stuff back ,although i had a good explaining to do to the big man himself as to why he could not log back in the network.

Simple really…

In reply to As a newbie at Tech Company i once worked for

At our office, the common response to “I have no email!” is, “Do you still have a job?”

Most lower level folks get a kick out of it… Higher ups unfortunately do not have such a sense of humor.

Power supply

In reply to As a student,

My first job out of university, taken on as a developer for a large logistics company and the IT Manager for some reason told me to start ripping out all the old cabling! Suffice to say his instructions, ‘to pull and cut all the thick cables’ led to numerous calls to the helpdesk from irrate users. To top it all I also managed, whilst cutting cables in the comms closet, to lean back onto the main power supply and shut down power to the whole building!

I left after two days and put it down to experience!

Am i the only one to do this?

In reply to As a student,

When i first started in IT, i came across a server that was running out of disk space, exchange would not run…So, i deleted loads of LOG files….including all the exchange server log files…DOH!!!!! Hence, i have never done it again….

I’m no Sage.

In reply to Am i the only one to do this?

I once deleted our entire accounts data after recovering everything from a damaged server after a flood.

We got all our data back and I copied everything over to the other server fine. The Sage install lived in a folder of old files from the previous server. After rebuilding our file system and copying over everything to the new machine, I “forgot” that I’d patched in our accountants to the old data in this temp folder, even though it was newly installed. Having spent a few weeks working through our backup system and checking everything was fine, I cleared out the “spurious” data. Then swore quite a lot when I got the first phonecall.

Not mine but a good-un

In reply to I’m no Sage.

Just the other day a close friend told me she had gone to Best Buy to purchase a new scanner. She’s a teacher and wanted a scanner that had OCR software. Upon describing what she wanted the knowlegable sales guy AND the “Geek Squad” guy BOTH told her that the technology didn’t exist.

She bought a scanner anyway and found the OCR software in the box when she got home. So much for knowldgeable staff and consultants.

Blow up a power supply today

In reply to As a student,

You are not alone in blowing up power supplies. I did the same on my own pc. Cant for the life of me remember what prompted me to do it except that it was there and I swear was winking at me.

On reflection though, its a pretty poor design that cant detect the correct voltage and ajust accordingly.

Knew of an electrician that installed an emergency power supply for four AS400s. He decided to demonstrate just how ‘uninteruptable’ it was by cutting the mains supply in the presence of the IT director.

Nothing happened of course, and a number of things stopped happening, including the four AS400s.

Story goes that the director chased the electrician around the site with a very real threat of violence.

Its a small world

In reply to As a student,

A guy in my dept. at work did the same thing during the replacement of one of our small file servers.

I can tell him this..maybe he’ll feel better.

An ancient history gaffe

In reply to As a student,

In my first programming job, I was maintenance programmer for a major steel company’s hourly payroll (about 250,000 workers’ checks”). It had a habit of blowing up about every other pay run, in which case the programmers had to fix bugs in time for the money to reach the mills the next day. If it didn’t, the workers had the right to walk out at 12:01 PM — while management walked all over the programmers’ heads.
Note that source code was kept in boxes of punched cards — about 2000 per box — of COBOL code. Absolutely necessary for rapid recompile and link.
So of course, one lonely Thursday afternoon before payday, yours truly pulled out the Gross to Net program deck, and began making some minor changes without first duplicating the deck. I set the box on a nearby wastebasket, which was already half full of miscellaneous card waste.
Sure enough, 5 minutes later I bumped the box into that wastebasket, irretrievably fouling the production source deck. There was NO backup. Additionally, all the (very fast) keypunchers had left for the night — there would be no help from that quarter, either.
What ensued was 6 hours of frantic punching AND verifying of a complete listing for that program, followed by a card printout and sight verification which took the rest of the night. Not only that — we were (unbelievably!) paid overtime in those days, but I couldn’t TELL anyone! I could only hope that no bugs developed that would require maintenance to the code — because it would have been fatal hours before anything could’ve been fixed.
Thankfully, no bugs showed up that night, and the deck was put back into service with no one the wiser. There were some suspicious looks directed toward my bleary eyes the next morning, but at least I HAD a job to go to — much better than the alternative!

RAID systems protect servers?

In reply to What’s the worst mistake you ever made as an IT pro?

I think the biggest gaff I made was when one of our customers purched a new external RAID box to go on their Exchange Server. I was asked if I could go in and configure it for them, I said no problem, I will pop in at the end of the day and set it up. After plugging in the external box and powering up the Compaq server (they are nice and easy for configuring RAID) a message appeared on the start up screen that the system had found new drives and would I like it to initialize these disks. I pressed “y” then immediately thought “no I don’t”. Too late, I had initialized the whole RAID including the system. The first indication I had that the customer had come back in the room was, “Does oh bollox mean that didn’t go quite right?”. I then spent until gone midnight re-installing windows NT4 and Exchange Server and restoring all mail etc. from the previous backup, which fortunately I had done just before plugging in the new box.

Ghost is brilliant

In reply to RAID systems protect servers?

Next error, the MD of one of my customers had a dodgy hard disk in his desktop pc, so I got a new one of exactly the same size and thought I would ghost his original system to save the time of re-loading th OS and re-installing all his software. Then, promply overwrote the original drive with a blank one! Fortunately no data saved on original drive, so did end up spending the time re-installing everything.

ERD Commander

In reply to Ghost is brilliant

Works great for mistakes just as you discribe. Which I have made on more than one occassion.

More Ghost Toast

In reply to Ghost is brilliant

I did something simular the other day. I had 3 identical machines to build for a customer and figured I’d use ghost to speed things up. I built two simultaniously and went to ghost the 3rd HDD, and oops, ghosted blank drive over imaged one. Turned out HDD was on secondary IDE Bus. Oh well I thought, still had one machine with a good image, but I forgot I moved the imaged HDD back to Primary Bus, and as I pressed enter on Ghost I realized I just asked to Ghost secondary drive to primary.
3 Hours to image the machines, 3 minutes to wipe them and another 3 hours to reimage again. I have since learned that it helps to use a spare different sized HDD to differenciate.

Ghosting Tips

In reply to More Ghost Toast

Although we may have the Ghosting process outlined in our heads, it’s still important to write down the Source and Destination Drives. A simple distraction by a co-worker (or a family member, if you’re doing this at home) could well throw you off. As a rule, always name your volumes clearly. For example, if you partitioned a Western Digital 300GB drive into two volumes, you could call the primary volume WDPrimary and the extended, WDExtended or whatever names you find logical. If you make three partitions, include the volume name — you could name them: WD300C, WD300D, and WD300E. The new drive however, must have no volume label or have ‘NEW’ in its name for easy recognition if you are restoring separate partitions — NEW1, NEW2 etc.,. After the Ghosting is complete, the new drive?s volume label will be overwritten, this is to make sure the intended source to destination direction is correct.

Reply To: What’s the worst mistake you ever made as an IT pro?

In reply to RAID systems protect servers?

Teaches you the value of backups….
It has to be tried befor you really understand how usefull that hour while backup is running is.

Totally Agree

In reply to Reply To: What’s the worst mistake you ever made as an IT pro?

I totally agree about backups, I am always on at my customers to do this. But who was the one that lost all his data after a hard disk failure with no backups, yup, me!

Backups definately save me!

In reply to Reply To: What’s the worst mistake you ever made as an IT pro?

We used to have a policy that email did not get back up, period. The exchange server was backed up, but not the mail store. One day at lunch I decided it was time to update NT4 to SP6. It was a relatively simple thing, so I figured I’d be safe. The service pack applied just fine and said it had to reboot (as is always the case with Windows). This is where my nightmere began. The server refused to come back up. It was start loading NT then blue screen long before it was actually up.

Not having a backup of the email, I really didn’t want to just do a tape restore as that would result in loosing all the email history. It was after a few failed attempts with command line utilities that I remembered that we had mirrored disks. So I broke the mirror and physically removed one of the disks. I then did my tape restore, put my mirrored disk in the server as a “d” drive, and used a command line utility (ERD Commander, as I recall, but that could be wrong) to copy the exchange databases over. I then took the “good” (yet really bad?!?) disk back out and booted up. Fortunately, that worked as the next step was to loose everybody’s email.

Shortly after that incident, coperate changed their policy and we now backup mail once a week.

been there

In reply to RAID systems protect servers?

did the same thing with a dell server at a bank. wiped all the check imaging. performed a restore to fix it. took two days. the bank manager just laughed at me. he was very nice about my screw-up. he still needles me to this day. he said “it is good to know that you are human.” we could use a few more people like him in th world.

I made the same mistake

In reply to RAID systems protect servers?

The RAID initialization burned me once as well, they should really call it a format.

I was adding a disk to a RAID card and ended up initalizing the new disk along with the disks the server was running. This was the centerpiece of the business. Backups saved me; however, there was not any docs on how to do a full restore of a domain controller, exchange, and veritas (all of which resided on this server). After seeing the sun rise and no sleep for 20 hours straight the system was back online. I am glad it happened to me, I was getting a bit cocky with my abilities and this disaster I caused put me in check.

I always have a second pair of eyes available when working on such things now. That one sure beat the hell out of me.

Lost a day worth of data

In reply to RAID systems protect servers?

It wasn’t my mistake but we lost a complete days work:

We used a product called Standby Server for Netware 4.11 back in 1999. You could hook up two (different) servers and Standby Server made the disks in the second (standby) server available as local disks so you could create a mirror.

Everything was working fine but we were having some performance issues so our vendor decided to upgrade the NIC drivers. We were asked to break the mirror in the morning so they could send over an engineer who could upgrade the drivers at the end of the day. After he upgraded the drivers he recreated the mirror but did it the wrong way around. The data on our production server was overwritten by the 8 hours old data from standby server. Of course there was no backup because he simply had to update the drivers for the NIC’s.

Domain Controller

In reply to What’s the worst mistake you ever made as an IT pro?

Hi,

I managed to crash the Main Domaincontroller at my previous work. It wouldn’t have been a problem
if there were a secundary DC, but there wasn’t. I was installing a printer on the machine and during the install i got a nice Blue Screen. I can tell you, you completely freak out when this happens. The other employees didn’t mind so much. They were all forced to go home early :P. Lucky for me, this happened at 3pm. I was able to fix the problem and was home at 9pm. I was also lucky to find the reason for the BS, so i could defend myself against me angry boss. It was just a matter of time…

The Future of PC Operating Systems

In reply to What’s the worst mistake you ever made as an IT pro?

I was a green salesman back in 1982. The IBM PC had just been launched. My product was the MDBS III database application. After a telephone marketing campaign, the first client turned up. Because the database application was sold in two flavours, the big question was which one of them should the client buy.

I was put on the spot – after a few umms and errs, my definitive recommendation was “…CP/M, because it has been around for a while and we are all comfortable with it, and we have heard that the newcomer, MS-DOS or whatever it is called, requires 128KB of memory to run.”

There you go…

Reply To: What’s the worst mistake you ever made as an IT pro?

In reply to The Future of PC Operating Systems

I will admit I have made a few.

Just last week I spent a while updating all the service packs/patches on an xp home machine.

It was an ASUS notebook and it had a shortcut on the desktop that said NTFS conversion or something like that.

Without thinking twice I thought “yep thats a good idea” and double clicked it, at the end of the batch file it said do you want to update the pointers or something like that.

A orange light was dimly flashing in the back of my brain and I knew I had a 50/50 chance, bugger it, I selected no.

Restarted the Notebook “please insert system disk”

2 hours worth of work down the drain.

Another crushing stuff up I have made is ghosting what I thought was the source disk (but was actually the blank destination disk).

I ended up with a perfect copy of a blank hard drive.

I did not stop the Stock Exchange, somebody else did…

In reply to What’s the worst mistake you ever made as an IT pro?

A couple of years ago a IT technician found the time wrong on a server at the stock exchange clearing center in our country. He set it back one hour(he thought) but instead he set the time back by one day. The result: When all the broker systems went in to collect their data all transactions from yesterday had disappeared, all starting prices for all stocks where from wednesday, instead of thursday, and people started to trade like maniacs since thursday had been a really turbulent day. After a while the trading was stopped (panic button :-)). The stock exchange start the trade again late in the afternoon. A whole day of trading missed by a simple mistake by a IT technician.

Deleted the Administrators

In reply to What’s the worst mistake you ever made as an IT pro?

I am responsible for the Student Ldap at a mayor University in Sweden. One time when I tested a new ldap browser I decided to delete some System Administrators that had left the University, a normal update of active Administrators. I marked the account in question and clicked on delete. Wrong place to do that! The entire folder containing the Systems Administrators for the University was deleted.

I had to learn in a hurry how to create an ldif from our secondery master over the deleted folder before they synchronized their records.

Worst Mistake

In reply to What’s the worst mistake you ever made as an IT pro?

…

I’ve made a few…

In reply to What’s the worst mistake you ever made as an IT pro?

and I have learned many VALUABLE lessons. Here are some of my worst:

1) Wrote an interface for an e-commerce site to their CC gateway and because of the way I chose to check the cards, if the card was good, but the address or CVV2 failed, it would deny the order but keep the hold on the card. The client sold items worth $2000+ and the customers just knew their card was good, so they kept trying, some as much as 5 times. This resulted in their cards having several holds put on them for several thousand dollars each.

2) I was asked to recover a web server hosting 100 domains from a hack attack, with no backups and no records of passwords, domains, clients, or email accounts. Well, all the executables (grep, ls, ps, etc…) had been corrupt, but they had an identical box, so I decided to copy the executables from that one to the other so that I could at least export all the data I needed from their control panel. Unfortunately, I copied the data from the other box too.. Lost all logins and email accounts, but was at least able to save all the sites, then came the process of calling each client and resetting all their account info, which is now logged.

3) Last, accidentally typed “rm -Rf /” on a production server when I was trying to press the right shift key on my laptop and accidently hit the enter key.. Now all my remove commands are typed into notepad first, then copy and pasted into my terminal window after ensuring accuracy..

We all make mistakes, if you do, don’t panic. Ask yourself what went wrong, then ask why did it go wrong. Learn from the mistake, and ask what you can do differently to prevent the mistake from happening again.

the dreaded rm -rf

In reply to I’ve made a few…

I was waiting for one of these! Not quite as bad, but I was troubleshooting a backup problem on one linux server, while I had a terminal open next to it of the main server that ran the backup program. You can guess what happened — I deleted the entire backup program. Of course, we had been backing up many things, but NOT the backup program configuration.
It took me the better part of a couple of days to recreate the entire backup configuration. You better believe the first thing I did was set a backup for that config file!!!

Reply To: What’s the worst mistake you ever made as an IT pro?

In reply to What’s the worst mistake you ever made as an IT pro?

Anybody can make mistakes. To really wreck havoc you need access to an IT system…..

As a junior programmer in the late 60ties I had a boss with a temper.Programming language was assembler – everything was numbers, and lots was binary code. It created very small and efficient programms, a big contrast of the cut-and-paste, select checkbox programming of today.

Anyway I couldn’t make out why I sometimes got a leading CMC7 ‘5’ digit. CMC7 was a ‘font’ used for magnetic reading of bankchecks, and shouldn’t be in files as this was a ‘printing only’ value. I asked my boss, and he went off on a rave. No help there.
So what do an eager young man wanting to move forward in the world and higher in the payroll do? Make a test just before writing the field, and remove the CMC7 ‘5’ if it’s there. Easy stuff, solved the problem immediately.

Accounts then started to behave mysteriously, investigate. Problem was found to be that the CMC7 ‘5’ digit was used internally for negative, unless you were calling the printing routine, where it was used as a digit. I had been writing all negative numbers as positives when updating fields, and subtraction and addition was the same thing.

Needed 3 months of nightly reruns while trying to catch up with real production.

Morale: If somebody asks for help – give it. There’s NO stupid questions – only stupid answers.

rgds
Peter

Stupid Question

In reply to Reply To: What’s the worst mistake you ever made as an IT pro?

There is not such thing as a stupid question. There are really no stupid answers either, there are wrong and right answers.

But the only stupid question and stupid answer is when neither is given.

I have done all of them.

Playing tricks

In reply to Reply To: What’s the worst mistake you ever made as an IT pro?

Some time ago I’ve created a batch file called f$#$.bat that answer “F$#% YOU!!!” when you write the word on the command line. That worked when I fell frustrated with something.
After some time I’ve found one of my coleagues completely crazy, running all kinds of anti-viruses and anti-spyware because he was mad with his programming and wrote f$%#& and the computer answered back.
We got good laughs when I explained him

I have advised HQ what is to be done…

In reply to What’s the worst mistake you ever made as an IT pro?

I have dared to very politely advise our HQ to reqire contractors rework the “Users Guide” they have written to “Guide to operators” and have even dared to apply an examplary TOC aiming actual activities in detail. Good Lord! So sorry…

How the electric tram upset a network

In reply to What’s the worst mistake you ever made as an IT pro?

In the early 80’s the company I worked at that time installed one of the early Ethernet networks (remember the thick yellow cable?) in The Hague (The Netherlands) The customer wanted the network to be installed in the weekend, so that normal operation could be continued on monday with the shiny new network! So we went and installed the network, tested it thoroughly, and left to enjoy the rest of the weekend (it was a fairly small network).
On monday the account manager called the client to ask how he liked the new network. He said fine, but every quarter of an hour or so, something odd happens, and then we cannot use it for about two minutes…
So we went back trying to find out what went wrong. Everything seemed to be in order, so in the end, desperately, we hung a scope onto the backbone in order to measure the signal. And indeed approximately every 12 minutes we saw an enormous amount of spikes rolling over the network…And we didn’t have clue as to where it was coming from….
As the day proceeded, we trying to figure out frantically where this signal was coming from, and, as it was a summer’s day, the temperature was quickly rising, so someone opened a window to cool things down a bit.
The sounds of the street were coming through the open window. Suddenly we heard: “Dingelingeling”, and over the network another series of spikes was sent, disabling communications over the network again.
Most Dutchmen know that when they hear “Dingelingeling”, the electric tram is close by and is warning you to stay out of its’ way.
It took us three “Dingelingeling”‘s to figure out the connection between the EMC field the Electric tram created, and the spikes on our first Ethernet network…

My biggest mistake? – not vetting clients

In reply to What’s the worst mistake you ever made as an IT pro?

I took on a project for a firm of lawyers – nothing too big, an exchange server, six client pcs, a wired LAN, and a couple of printers. The staff in the office were basically computer illiterates, calls ranged from “I clicked in “record a macro” and don’t know what I’m doing” to the final straw when I got a telling off over the phone at seven thirty one morning because one of them couldn’t get a document to print. Turned out the printer wasn’t switched on. After that gem, and no apology, as it was obviously all my fault, I told them to find someone else, and now check who I do any work for before I start, and have a hefty call out charge (?100.00) for spurious and frankly stupid calls.

Local Executive Housing Franchisee

In reply to My biggest mistake? – not vetting clients

New server, upgrade OS, upgrade Eschange.

Except the owner didn’t have time for “that project management nonsense” and his wife, ever the Cardassian-style “leader”, thought her every modification to the scope would somehow reduce the contracted price. This included, but was not limited to, upgrading her OS, upgrading her version of Peachtree, reapir of an aging workstation, and adding it to the network.

We should have known immediately after the contract was signed, and the initial payment was called for, and the owner said, “well that’s what we agreed to, but you didn’t really think we were going to give you an upfront payment, did you?”

Blown power supply, revoked rights form administrator

In reply to What’s the worst mistake you ever made as an IT pro?

My first unforgetable mistake was installing a new pc at one of my freinds home, that had 240V and 120V power supply switch at the back, the switch was set to 120v, which I didn’t checked and plugged it into the mains of 240V….bugger I was so embbarresed….
Then recently I was creating a restricted user for windows NT on my work PC…but went into trouble creating one…so I though why not just revoke some folders right from administrator and allow the person to use it for a while….and then I revoked the rights from administrator profile and couldn’t logged back in the machine…

Then guess I ended my night rescuing my files for which I didn’t had backup and installing NT from scratch.

XP? its only a client….

In reply to Blown power supply, revoked rights form administrator

I was a workshop tech with a small ISP. I built myself a nice new machine under an agreement that it would be used to sell more of same. It was the first XP Athlon in the area and was fire engine red! I loaded the latest MS offering, XP Pro on it, no problem boss it’s just the latest NT client. The moment I plugged in the ethernet the NT4 radius server and XP had a spat about who was boss around here and the NT radius server retired to sulk. Not only would it not talk to XP, but it wouldnt talk to the 800 odd w95 & w98 clients out there either….ooops! I learnt a lot that afternoon in a diverse body of fields, client communication, discreet apologising for beginners, and the guts of radius servers and ISP networks. Ouch!

Classic SQL Error…

In reply to What’s the worst mistake you ever made as an IT pro?

I had to go and beg the local DBA to restore from a backup when an application was discovered to terminate with extreme prejudice if a user called up a record with an apostrophe in it. This, however, was not the problem, my interim fix was:
update tbl set myfield=”Great `O` Rings”

Because I forgot the Where clause I had that feeling of great doom as my SQL took more than a fraction of a second to run…

Been There, Done That!!

In reply to Classic SQL Error…

I’ll never forget that feeling of doom…
Thank god the table had a couple million rows and I had time to yank the network cable out of the workstation then kill the process from the admin app. Took a while to roll back but I had to go for a walk outside and contemplate a career change.

Thinking the Division Head meant it when

In reply to What’s the worst mistake you ever made as an IT pro?

he said ‘This is an open organisation speak up if you see any problems with this change in operations plan and budget’. Wrong

The whole new idea was put together by my boss an old time network design guy – his last network was coax. The Division Head, his boss, was a number cruncher with some basic IT knowledge. Neither new a thing about designing modern gateways.

The new plan was to lever a new partnership by selling dozens of new appliances (intergrated hardware and software that did routing, anti-virus and web proxy) to our existing clients. The intent was to just remove the existing gateways and put these in place when upgrading as they were cheaper than a the usual PCs with software, and quicker to put in place. Funding was included to replace some at critical points at out expense as we could remote manage them better.

Erroneously I believed the big boss’ statement and told him that we should factor in a thousand or so for each install, to pay for a needs analysis and suitability report on the system as the new appliance was very useful IF the gateway was set up in a particular way – if not you had to redesign the gateway to suit the appliance as the unit was not adjustable in any way.

Neither undrstood gateways Big boss spoke with my boss, they spoke to marketing man from partner who made appliance – he had no IT knowledge in this area either. result I got laid off and a year later they both got no bonuses as the project ran over budget by 45%, over A$200,000. Seems only one system would work as a straight swap of the unit, the rest had major down time as they were urgently redesigned.

A coworker had kept a tape of the meeting where I spoke up, that got th bosses into trouble for not properly investigating my claims.

My biggest boo boo

In reply to What’s the worst mistake you ever made as an IT pro?

Allowing the companies I worked for to label me as, and utilize me as an expert in a specific software package. As technology moves, companies rely on the expert to maintain the legacy application, often denying requests for training on new technology. When the company moved to new technology I was too expensive to retool or retrain. Younger talent was cheaper, and oftencame with the knowledge necessary to jump right in.

zzzfor years I was the person on the staff that kept things going. Now I’m a decade behind the times, with a lousy choice. Retool and retrain myself, and pray for another company to give me a chance to start at the bottom again, or change careers.

12 boxes of paper

In reply to My biggest boo boo

I submitted a mainframe (you remember those?) print job on Friday evening just before leaving the office. In the shower on Monday morning I remembered (‘remembered’ isn’t really strong enough to describe the lump in my throat) that I forgot to sort the input file by department.

You see, the report did a page break on department – so instead of having 55 lines on each page it created a new page every one or two lines.

When I got to work I walked into the computer room and told the operator to toss the twelve boxes of fan-fold easy-read paper that had my name on them.

I want to thank Joe (his real name) for never telling anyone about this rookie mistake.

Melted $100,000 (2006 dollars) of expensive core memory

In reply to What’s the worst mistake you ever made as an IT pro?

Waaaaaaaaay back in college (1972), our college ran an IBM 360/48. (In those days, that was a mid-range machine worth about 1/4 million. With 48K (miniscule by today’s stanmdards) of core memory, the CPU alone measured 6′ x 3′ x 8′.)

Late one night while, when all our batch processing was finished, I was experimenting IPL-ing (booting) the machine from different devices (card reader, tape drive). I was intrigued how the machine would respond when I tried to IPL from a disk that was not powered up.

I hit the IPL button. Within seconds, the machine shut down. All lights on the operator’s panel were out except for one incredibly bright red Thermal Check light.

I opened up the access doors and found a reset switch. I hit it and the light went out. I then hit the main power button. So far, so good.

THIS time, I powered up all the disk drives and hit the IPL button. (No more screwing around – I was sweating bullets.) I crossed my fingers and hit the IPL button. Again, the power flipped back off and the Thermal Check light was the only light glowing were there chould have been about 100 happy lights blinking furiously away.

Gulp! I thought that was the end of my college career.

IBM was called and a crucial 8K unit of very expensive core memory had to be replaced. Instead of a nice array of neatly arranged ferrite cores, the Customer Engineer found a puddle of melted ferrite and tangle of burnt wires.

8K or core memory cost $24,000 — 1972 dollars! (That has to be equivalent to something like $100,000 nowadays.)

IBM replaced the memory under warranty. (Let’s face it, it shouldn’t have happened. But then again, what kind of idiot deliberately IPLs that a machine that way?)

I was lucky. Had I not been doing a valid project on Operating Systems design, I might have ended up as a plumber*.

* Making $80,000 a year with steady employment! Maybe I should have been thrown out!!

One of Two

In reply to What’s the worst mistake you ever made as an IT pro?

I’m not sure which was worse.

Back in the 80s I worked at a Satellite campus of a University in California. There was no manual or training for using the comupters, my trainer told me (over the phone) to press each F-Key, then Shift-F, Alt-F, Control-F and write down what each one did.

One of those combination put our server into a looped diagnostic function effectively shutting down operations at our campus and one other. ITD spent a couple of days tracing the problem before they realized what had happened.

More recently, ITD had called to complain that our Department’s server needed to be paired down, evidently 50 Gigs was too much to back up every day. I went into Groupwise to see what the defaults were to I could talk with the Boss about reducing some files. When I exited, I hit “RUN” not “CANCEL” and initiated a purge of 60+ data resulting in a 35 Gig reduction on the Server.

It took two weeks to merge the backed up data back into the Supervisor’s mailboxes.

No cash for you this morning

In reply to What’s the worst mistake you ever made as an IT pro?

Monday – Began a server consolidation project within a mid-sized national bank head office after months of planning.

Tuesday – Broke Nothing…

Wednesday – Server Farm Analysis – All good.

Thursday – Regional Bank Branch Logon Script Analysis. ?What?s this F:\ in the script??? There’s no logical mapping to an F:\ drive anywhere – dump it”.

Friday – “Eh, why are all 39 branches reporting off-line???” Restored script, reboot all regional servers…. 2 and a bit hours.

Saturday – Blinding Hang-over – angry wife.

Cost of project: $25,000
Cost of Alcohol: $50
People without cash for nearly 3 hours: ~800
Banking somewhere else – Priceless!

Hum Fixing The AR system

In reply to What’s the worst mistake you ever made as an IT pro?

Onced fixed a BT BACS system and collected 6 months of over duepayments.

Only problem is put a “Name” Jounalist over his overdraft limit.

Never got a thank you from The accounts department for recovinging 3/4 of a mill though.

Blue Teeth

In reply to What’s the worst mistake you ever made as an IT pro?

I go back a few years in the industry and back in the good old days when everything was serial connections you did a lot of stripping of cable. Sometimes you got into the bad habit of stripping with your teeth. To make a long story short one day I was stripping wire and bite down good only to find out the wire was run along with 120 wiring and got a nice mouth full of 120v at 12amps. Never again used my teeth.

Reply To: What’s the worst mistake you ever made as an IT pro?

In reply to What’s the worst mistake you ever made as an IT pro?

In the process of upgrading a Novell 4.10 server, I wiped it out. It took an entire day to restore the server. The most embarrassing part was that I was in the midst of studying and taking exams to become a CNE. I took the final test 6 days after trashing the server. I didn’t exactly trumpet that achievement to anybody on my team.

To this day, I have no idea what happened.

Maybe not your fault but…

In reply to Reply To: What’s the worst mistake you ever made as an IT pro?

Even though Novell had a great rep as a technical vendor,I have been in situations where EVEN if you do everything by the book,
SH!T HAPPENS.

Been there done that.

I have had a lot of issues with Netware upgrades due to flakey software tools.

I did a DS upgrade that hammered one of my servers DS database.
I had done 5 or six the exact same way.
I called Novell and they said ,yeah this happens occasionally.

Gee ,thanks for the warrning!

Were did all of the accounts go???

In reply to What’s the worst mistake you ever made as an IT pro?

I was doing security cleanup work with a team of people. Anyways, One of the tasks that we had do was to delete unnecessary LOCAL accounts from member servers in a NT 4.0 domain. Well, one of the guys that I was with, logged into the NT 4.0 PDC, and thought he was looking at LOCAL machine accounts. He ended up deleting everyone’s DOMAIN accounts.

Needless to say we ended up with egg on our face, and had a heck of a time trying to rebuild the Exchange Server.

Tombstoned Computer Accounts??

In reply to Were did all of the accounts go???

During a NT4 to W2K3 upgrade, I was browsing the NT4 domain and noticed about 2000 “tombstoned” computer accounts, wanting to cleanup the NT4 domain I figure lets delete them, on Friday afternoon no less. Well saturday morning my pager was going crazy with people getting errors about the domain unavailable. Opps. these were remote workers using laptops. These users would login with a cached account, start vpn then logoff and then attempt to login, no good. Took us a few hours to get the correct process to re-add them. 3 weeks later we had all the legimite accounts back and the real “tombstoned” accoutns removed. Nothing like scaring the crap out of yourself…..

Reply To: What’s the worst mistake you ever made as an IT pro?

In reply to What’s the worst mistake you ever made as an IT pro?

There are 2 kinds of people. Those who backup and those who will.

IBM 360 “blunder”

In reply to What’s the worst mistake you ever made as an IT pro?

I was still in college in the ’70’s, but was working part time – night shift as a computer operator on a rather large IBM 360 system. The console was a vast array of switches, dials, and lights, which overwhelmed, but intrigued this computer science major. I decided, for some reason unknown to me to this date, that the system needed an “IPL” (initial Program Load – or a mainframe reboot). So, I proceded with this task, thinking that I had the knowledge for I had seen this done once before by my supervisor. Well, needless to say, I screwed it up, but I did not tell anybody at the time. A few minutes later, when the system did not respond to console commands, I had to fess up. I was majorly embarrassed, but my supervisor was easy on me since it was a slow night. Those kind of things stick with you.

PS. After reading other posts, at least I didn’t melt the memory. LOL

Trusting my boss…

In reply to What’s the worst mistake you ever made as an IT pro?

After developing a corporate wide system to evaluate projects and establish prioritizations my boss wanted me to train two younger staff on the process. The staff was trained and I was gone 6 months short of 20 years. The reward for hard work and dedication often comes too early and is given the wrong people. Over 50? Watch your back!

My biggest blunder I ever made during my IT career is…

In reply to What’s the worst mistake you ever made as an IT pro?

… having one in the 1st place.

Windows Lockups

In reply to What’s the worst mistake you ever made as an IT pro?

Had a client with an older desktop machine with lots of mortgage files. They did not back up, ever.

The PC kept locking up; random amounts of time. I fussed and fussed with it, loading service packs, etc. Finally, formatted and reloaded Windows but was only able to save a few things. The PC started locking up again after the reload. Turns out the fan on the CPU was stopping and starting at random intervals.

Someone else discovered this when they scrapped the machine. I should have looked closer at hardware; it is so easy ?to just blame Windows?. The client never paid me a dime.

Ooops the fiber broke

In reply to What’s the worst mistake you ever made as an IT pro?

The biggest blunder I ever made was when I walked behind a telephone system and snapped the fiber optic cable off. We didn’t notice it for about twenty minutes when we got a call that users were not able to make phone calls. Luckily we had a back up set and had it going in a few minutes.

Now the biggest one I ever saw was when my manager pressed the EPO button on the UPS to silence the alarm. EPO stands for Emergency Power Off and it shutdown the whole datacenter which took hours to bring back up. He swears to this day that he didn?t do it. About a thousand Lotus Notes users were down and about 300 desktops connected to the system. Not to mention the UNIX and Digital equipment that was providing the production systems information.

Costly mistake

In reply to Ooops the fiber broke

Those of you that know Sun servers will love this. The 4600 series and similar models have CPU trays that attach to a centerplane. I was putting one in once and didn’t notice it had a hard plastic dustcover on it’s connector. the guidepins on the centerplane pierced this cover, and then promptly came out with it when I removed the board to see why it wasn’t seating properly. Those centerplanes aren’t cheap, but luckily they were able to fix this one. Thank god this was internal at a production facility, or I would have been in a world of hurt.

another EPO story

In reply to Costly mistake

Working at the data center for a major S&L. They had a UPS (or some delivery company) guy drop off something. Rather then just hand it to someone outside the glass house they let the goober into the DC. The doofus saw the big red button and pushed it. That was in the day of mainframes (IBM 3083 and such) that just simply stopped. What a mess. We sued the delivery company and they sued us for being dumb enough to let the guy in. No clue who won since I left before it was resolved.

Similar type thing

In reply to another EPO story

Sitting in my office, I could look through a glass wall to our bullpen, through another glass wall to the’puter rom and through the far side to the hallway. I see the HR manager with some other guy at the far end, nd idlely wonder what that wand thing is. To my dismay, the HR guy pulls out a swipe & into the computer room they go. All of a sudden I realize IT IS A SMOKE WAND. The numskulls were testing the firealarms with a smoke wand. I ran & vaulted a desk & tried to yell, but yes, the alarm worked & just like it was supposed, it killed all power to the room… the mainframe, the Unix servers, the Novell servers, the AS400 and the telephone switch.

I took the card away from him.

I could write a book on blunders

In reply to What’s the worst mistake you ever made as an IT pro?

I have made quite a few “oops” in my time. I have learned from all of them and not made any twice. I think the biggest blunder I made was while designing and installing a complete new network for the company I worked for, I told my boss how easy it was going to be. I installed all the switches and routers (luckily parrallel to the existing network) once it was installed we started moving things to the new network to find out nothing worked. I forgot to turn on IP Routing……DOH!!!!

Won’t bother backing it up – it will b ok!?

In reply to What’s the worst mistake you ever made as an IT pro?

A few years back whilst working for a large firm of housebuilders, the accounts managers pc was problematic.
Staying on after office hours I decided that the problem lay in a malfunctioning hard drive.
The pc was a two drive system with os on disk 1, and data on disk 2.
Didn’t need to back up disk1, just reformatted & reinstalled os.
Disk 2 was not being worked on so decided not to bother backing it up.
Os booted fine, but to my horror data drive 2 had gone t**ts up for no apperent reason.
Never mind – will restore from accounts backup in AM.
Turned out most recent tapes they had taken were over 4 months old, and I had succeeded in wiping out 4 months worth of accounts info.
Lets just say i did not last very long after in their IT Dept, and am now a compulsive backer upperer!

When I learned the meaning of “KISS”…..

In reply to What’s the worst mistake you ever made as an IT pro?

….”Keep it simple stupid”
I had a laptop with a black screen. After a few hours of dis-assembling the damn thing and putting it back together I still had no idea what was causing the problem. I then noticed a little slider switch on the side of the monitor and when I slid it up I had “solved” the problem.
I explained to the end user that the “video adapter had to be reset and the precision refractometer had to be recalibrated”. He beleived that story, which proves my second theory “If you can’t dazzle ’em with brillance then baffle them with bull#@%&”.

Device Drivers are important

In reply to What’s the worst mistake you ever made as an IT pro?

In Unix, you can use the find command to also perform other commands. It saves a few keystrokes. I needed to find and delete several files. I carefully crafted my find command, specifying the directory to start with, the parameters of what to delete, etc. I was running it from the root directory, logged in as root. Pressed Enter and got suspicious when the command took more than a few seconds to run.

Killed the find and discovered it was deleting everything, starting in the /dev directory, the repository for all device drivers.

The system was still up, presumably because the drivers it was already using were loaded into memory. I wanted to restore /dev from tape, but the tape drive driver was gone. In a panic, I decided to reboot, totally forgetting that things like the console and keyboard also relied on [missing] drivers in /dev. Then everything was dead. I don’t remember what I did to get the system back up, though it required me to be very clever and involved booting from the install disks.

After that, I started testing all my potentially dangerous commands in a sandbox.

I’ve got a couple

In reply to What’s the worst mistake you ever made as an IT pro?

As a fairly green mainframe field engineer, I was re-installing a system in a new facility. I got everything working except the A-D controller, and for the life of me I couldn’t figure out why it wouldn’t work. I finally swallowed my pride and called for help. They flew in the A-D guru from the coast, and after about fifteen minutes he found the problem: I had forgotten to connect a pair of cables under the raised floor. Fortunately the guru was sympathetic, and reported the fix as “repaired open circuit”.

Another time I was troubleshooting a tough core memory problem. I had several double-layer memory boards stacked on the floor when I stepped back and crunched one of them under my heel — $2800 each in 1970. I tagged it “bad from stock” and sent it back with my fingers crossed, but I never heard anything more about it.

I worked for another mainframe company that stocked customer sites with board testers, the idea being that we repaired circuit boards on site rather than swapping them out. A high-profile customer called on a Saturday and I went to fix the problem. I started testing likely candidates, but every board I tested showed errors. This went on for most of the day, one frustration after another. When the night engineer came in to help, he immediately saw that the corner of a plastic bag that a board came in was stuck in the tester socket, ensuring that every board tested would fail!

It Balanced

In reply to What’s the worst mistake you ever made as an IT pro?

It was 1972 and I was working as the lead night operator at a bank (before branches). Each day, four runs of checks and deposits were processed on a huge MICR reader. about every 20-50 documents a separator with a subtotal was inserted to aid in finding errors from unreadable checks.
After running the last run for the day, I asked the junior operator to verify that the totals balanced. When he confirmed that the amounts were in balance I gave the go ahead to begin sorting the paper checks by account number for filing.
He started sorting ALL the checks (which also stacked the separators away from the check). Only after the second pass of sorting did I discover that what balanced was only the last entry run; he had followed the beginning of the day process to clear all data entry files as he started the last run.
After thinking, I started reloading all the checks, which were, by now, one huge batch with no subtotals and with many of them shredded by the check reader. Fourteen hours later (at 8:00 AM) I had to explain the management why we had many thousand of dollars missing from the checks posted to customers accounts (we had already paid the checks) and no subtotals to identify which portions were out of balance.

Deleted Active Directory

In reply to What’s the worst mistake you ever made as an IT pro?

I had just went live with a large user provisioning project across all our major platforms. The Active Directory data did not sync properly with the central data store. I decided to clear the contents from central stores database for all AD platforms and then re-download.
Unfortunately, I forgot to stop communication to the remote platforms. I ran a simple SQL statement (Delete * from [datastore] where platform=AD). Dutifully and as designed, the system begain removing each of the 40,000 user and computer accounts from all AD domains.

Luckily, I realized the communication was live and stopped it just before it deleted the CIO’s account…

external harddrive

In reply to What’s the worst mistake you ever made as an IT pro?

whilst working for a university architectural department as technical support, i was given a external harddrive by a member of staff,

diagnosed the problem as a distinct lack of partiton information and set off a standard data recovery program.

whilst the harddrive was being scanned and the program happily reporting recoverable data, i’d thought id leave it to finish scanning.

unfortunately the powersupply cable some how got tangled in with the chair leg and when i moved to stand, the chair tugged the cable and the drive proceeded to get flung floorwards, then hitting said floor with a sickening crack.

it looked ok but when i plugged the drive back in, all i could hear was a high pitched scaping sound and no reporting of any drive connected.

two years of documents that was on the disk were gone.

They said it was idiot-proof

In reply to What’s the worst mistake you ever made as an IT pro?

Early in my career, I was duped into working on a user’s home machine. For some reason I got inside it and decided to pull out the CPU because I had never done that. So I put it back, turned it on, heard a pop and smelt burnt plastic. I had somehow put it in wrong! I was the idioit that broke the rule.

Trusting Management

In reply to What’s the worst mistake you ever made as an IT pro?

Managers have all of the bad traits of the worst
employees you have ever worked with. They lie and are egotistical, insane, and psychotic.
1. If a manager says hump out for me for two years and you will get a 25% pay increase–don’t believe it–this manager will be transferred/promoted and the new manager will say
“That has nothing to do with me!?”–Advice–Get your money going in the door and leave the promises for the newbies.
2. Managers of one IT company wern’t getting any new business in–sure enough the corp went belly up later–even if it took 10 years.
3. we backed up 1 database of 40,000 bytes and got 400,000 bytes–the manager said “load and use this database”–so we ignored his advice.
4. More later…as duty calls….

Promises Promises

In reply to Trusting Management

Promises are made to be broken. Have them show you the money or tell them to take a hike. When I started out 11 years ago I was ripped off by a consulting company and it caused me to loose my job temporarily. The scummy company I was working for sold me for 40 dollars an hour and paid me 15. They told them I almost had my MCSE and that I was engineer quality person. After about five months I got the boot when they found out how much the company was actually paying me. I was hired back at 27.00 an hour. My replacement was fired and I ended up being borrowed by the department I used to work for and used his build to do the roll out. Funny how things work out. He lied about having his MCSE.

Uninstalled IIS on web server

In reply to What’s the worst mistake you ever made as an IT pro?

I use a KVM switch to look at my servers from one console. I needed to install IIS on an old server to practice setting up SSL. I uninstalled and reinstalled IIS before I realized I was looking at the web server. Configuring everything to make the web site fully functional took several hours. I now have a graphic of each server’s name in a HUGE font size as wallpaper on each of them.

Blundering about….

In reply to What’s the worst mistake you ever made as an IT pro?

After more than 25 years in computers and networking, probably the one that sticks out the most is one day many years ago.

I was working to run a cable from the back of a Pr1me mini computer to the patch panel. The back of the cabinet was barely far enough from the wall to open the door, I was down on my stomach with my head and shoulders under the floor. (of course, the back of the cabinet was open, and the power supply switches are down in the bottom.)

You can see where this is going don’t you? In the middle of the day, a simple task, and POW!, I nudged the main breaker to OFF when I rolled over to sit up. The sound of fans and equipment going silent was terrible!

Got it back on quickly, but of course, the damage is done. About a 15 minute outage in the middle of the day. Boy, was I embarrased, and humbled!.

J.

What does THIS switch do?

In reply to What’s the worst mistake you ever made as an IT pro?

One day I was rattling around the switch and patch-panel racks
lamenting about my network administrators inability to use the
cableling system as it was designed. After running my monthly
diagnostic on the APC Backup-Power Tower, I decided to inspect
all of its connections on the back of the unit. Everything was
snug and I should have left it alone. But NOooo… I had to touch
the breaker reset lever. I barely touched it (I sware! :0) ) .. well it
tripped … killed the power to the entire unit. Suddenly the sever
room was VERY quiet. I waited a minute or so, reset the unit and
everything came back to life.

Unfortunately, the database server had a seizure due to the
number of people who were accessing it during its improper
shutdown. After running the database repair routines we
reloaded … but no go … the biggest of the relationals was too
far gone for repair.

Now at this point most of you are saying “reach for your
backup”. That we did .. unfortunately it was a week old. That
particular server had been rebuilt the previous week and the
reconnection to the nightly backup server had been missed after
the rebuild. I take the blame for that one to since I should have
double checked my guy’s work.

Now the sinking nausea of panic begins to creep in … but we
were saved by locating a recent ghost image of the troubled
database that was recent downloaded by one of our remote
offices a few days earlier. That reduced the missing data from 5
-7 business days down to around two days. Needless to say we
spent the next week re-entering missing data for our users as it
was located from other sources.

The next time you see a “Big Switch (any color)”, ask yourself ..
“Do I really need to see what this does?” … then walk away.

Or this one

In reply to What does THIS switch do?

Many years ago, I heard this tale of woe…

Back in the days of the big mainframe (like the one the fellow melted the core on), there was this switch on the front labeled “Emergency Pull”. It was intended to be used IF and when someone was ether being electrocuted or physically “hung up” on a moving part of the machine. What it did was drop all power to everything IMMEDIATELY. Nothing graceful at all.

The new machine had just been installed and the CEO was being given a tour. He wondered what the switch did and before anyone could stop him, he pulled it.

After three days of work by several IBM repair guys, the machine was back functioning.

DELETED!

In reply to What’s the worst mistake you ever made as an IT pro?

I suppose it could be the time I formatted the boss’s original floppy instead of the blank I was going to copy it to, but I’d have to give the nod to the time I was handling a tech support call, helping a user reinstall our application. This was back in the DOS days, and I didn’t make absolutely sure what directory he was in when I had him do the del *.*. I had to spend an extra 45 minutes walking him through how the undelete command worked to get his root directory back.

Accidentally removed IIS…from an Exchange Server

In reply to What’s the worst mistake you ever made as an IT pro?

I was in the process of locking down a newly build server. I was flipping back and forth on the KVM a few too many times and thought I had gotten back to the intended new server.

While surveying the server, I noticed that it had IIS installed and thought that there was no reason to have IIS on the box as it was not a a web server. I proceeded to uninstall IIS.

A few minutes later we had a lineup of users asking why email was not working. I quickly realized what I had done.

I was able to get IIS re-installed and Exchange repaired in only 2 hours. Thankfully, it went without a hitch and the system administrator could only laugh at me.

My biggest mistake

In reply to What’s the worst mistake you ever made as an IT pro?

Besides getting into the IT field in the first place?

Ooops! I shut down a mfg plant.

In reply to My biggest mistake

Back in the 70’s when I was a shiny new mainframe operator, I was working a weekend shift running jobs at a manufacturing plant. This was before any automated job submitter and the “schedule” consisted of a list of jobs written in a notebook with the appropriate input/output tapes. As the jobs were run, the operator was supposed to check them off. The schedule for the evening and weekends rarely changed and I’d been doing it for about 6 months so I was fairly comfortable with the routine.

I was running a series of jobs on a Sun. afternoon whose elapsed time was about 6 hours and whose outputs included thousands of pages of inventory reports that told various work units in the plant how much of what to produce in the coming week.

When I came in on Monday afternoon to work a swing shift my boss nearly had to fire me. Somehow I had skipped a job (even though I checked it off on the log) which resulted in not updating the files and consequently reprinting the previous weeks reports.

The most of the first shift at the plant (several hundred people) sat idle for most of the day while the recovery/rerun was performed.

I’m sure there were people further up the ladder who wanted me fired on the spot. But I think the operations manager really liked me and he understood that it was a manual process and simple human error is sometimes unavoidable.

I didn’t lose my job over it, but I was kind of treated like a leper for a while after that.

we all make mistakes

In reply to What’s the worst mistake you ever made as an IT pro?

Ok, I’ve done all sorts of things, from disassembling / reassembling a laptop and having leftover screws to formating a hdd before backing it up. The best one though, that I’ve witnessed, but not participated in, occured while I worked as a tech at a small computer shop. The boss brought in a dual celeron machine from a friend of his for the store to sell. The lead tech was curious and want to see if he could overclock it a bit. His first attempt put it beyond it’s capability and it simply wouldn’t start up so he powered down and found what he though was the jumper to reset the bios. It was about 2 seconds after he powered it backup that we noticed a nice electrical smell, he hadn’t jumpered the bios reset, instead he had bridged two pins on one of the motherboard fan jumpers, frying them all. Once reset the machine restarted fine, but none of the fans that were attached to the board worked, that entire set of circuitry was cooked.

Trust and Verify

In reply to What’s the worst mistake you ever made as an IT pro?

Worst mistake I’ve made in my career has was when my Windows 2000 server with Exchange 2000 on it BSD’d after installing a printer driver. Called Microsoft and the tech walked me through steps. After a couple of hours the tech said to hard demote the server and then promote it again. I said you can’t do that because the server was the exchange server and also the master FSMO. I asked him are you sure because everything I knew did not permit that process in windows 2000. Microsoft Tech said no problems and 6 hours later we used a backup to restore the server and 24 hours later the rest of the network was recovered. Backup! Backup! Backup! First… And if your gut says you can’t do that get another opinion.

Ever Make Users Cry?

In reply to What’s the worst mistake you ever made as an IT pro?

We have, and yes, actual tears. We were upgrading our Exchange 5.0 server to 5.5 late one evening. Along the good ol upgrade path there was the part that wanted us to perform a backup, so of course we told it go ahead and backup. well, during the backup there was of course an error, so we call MS and explain the error to them and they say, oh, don’t worry about that, go ahead and perform the upgrade. We did, and you guessed it, our mailboxes were no longer there, the users really cried more over the loss of their contacts rather than their mail, but they actually had tears streaming down their face. And BTW, we are not allowed, where we work, to back up exchange at all, never did and never will. Hopefully this is the biggest mistake I ever make during my IT career, because I don’t think I’ll survive another one like this!

We used to have that policy

In reply to Ever Make Users Cry?

We used to have the policy that the Exchange database was never backed up, period. That changed after I almost lost everybody’s email (story up above somewhere). We now backup “brick level” (meaning that only an entire mailbox can be restored, I think) and only one time a week on a tape that gets taken offsite to be overwritten in two weeks (rotating two tapes). We never have a backup that’s more than two weeks old of the mail server. That was considered a good compromise between the “no old mails that can come back later from some archive” and “we’re screwed if anything happens to the Exchange server.”

Automatic Update Mistake

In reply to What’s the worst mistake you ever made as an IT pro?

It was in the days of DOS. I was at a site that was using DOS 6.21. All the PCs were suppose to be DOS 6.21. They had been delivered from the manufacturer as being DOS 6.21. It was a requirement to the manufacturer that all the PCs use a single version of the OS.

We had 1200+ workstations, so, whenever possible, we automated things. There was an update to a program that we needed to distribute. The update could be simply copied on top of the older version to work (that was one of the really nice things of DOS … often much easier updates).

I took a base system, applied the update and then created an image using a program that compared files. The instructions given to the update script was to overwrite any older file with the newer file from the image.

Unfortunately, I left the DOS directory in the image and about 30 of the PCs were actually running DOS 6.20 (an older version). The update program could not rewrite the boot files, so the systems still booted in DOS 6.20 even though the DOS directory files were now DOS 6.21. When the update completed, the 6.20 PCs booted, but quickly generated incorrect DOS version errors as the DOS 6.21 files started to TRY to execute.

The real disaster for me was that one of the 6.20 PCs was on my boss’ desk. My boss, of course, was not technical and had no sympathy for the fact that the manufacturer was suppose to put the same version of the OS on every machine. Needless to say my pant size was chewed down a few levels that morning.

Using the same update image software, I was able to correct the problem on all 30 machines in about 90 minutes, but my boss was still pretty upset that I had caused 30 machines to fail.

Oh well. The next update took much longer and when my boss asked for an explanation, I reminder her of the previous disaster. Enough was said …

Power is everything

In reply to What’s the worst mistake you ever made as an IT pro?

My worst mistake was during a unix server failure. We had one of our production servers fail, and were trying to make rack room to work on the server, since our server room was crowded. I went to the UPS to unplug the failed server, and saw the label on the UPS at the plugs that said “Server1”. I did not trace the power cord back to the actual server, and just pulled the plug. Of course, the label was old, and the cord was actually a different production server. When I unplugged it, the databases dropped hard, and did not come back up. We now had both of our production servers down on the same day!
Lesson learned: Trace that cord before unplugging ANYTHING!!!

Newbie Notes developer

In reply to Power is everything

I was tasked with updating an old application that notifies our corporate office of IT outages. In order to save money and gain some Lotus Notes development knowledge I arranged a crash course on Notes Developement and tried to redesign the application myself.
My first challenge was to create an agent that would notify me when a field was not updated after 30 minutes. Some ameteur coding resulted in the agent emailing all of the corporate office every time it ran, as I had not specified how often or when the agent should run. The agent ran for 4 minutes before the phones started ringing, in that time we sent every user in our corporate office close to 1,000 emails.
: (

All females are not the same…

In reply to What’s the worst mistake you ever made as an IT pro?

I was (a long time ago) working in R & D in a company that made an expansion sound card. They issued several prototypes of the card to the embedded software developers. If you wanted to listen to the sound they provided a jack in the back where you could plug in a telephone handset. The hardware engineers apparently had a lot of keyboard plugs laying around so they just wired up any old handy female looking thingy for the prototype jack, as that was not how the final product would be anyway. Of course this made it plug compatible with the keyboard.. I bet you can see where I’m going…

One day my boss was on vacation and I needed something off his computer. I turned it on without looking at it first and was quickly unnerved by that familiar burning electrical circuitry smell. You guessed it. Someone had unplugged his keyboard and plugged it into the jack in the back of the prototype card where the handset was supposed to go. Unlike a normal keyboard socket, that sucker was delivering 12 volts. Pfffft!

DEU (Deffective End User)

In reply to What’s the worst mistake you ever made as an IT pro?

Many years ago when I worked as a tech in a retail shop I was working on a computer for a lady who said her computer would reboot while she was in Word. I had been working on it for days and couldn’t find anything wrong with it. She told me I had to be typing in Word for some time before the problem to occur, which I hadn’t really taken the time to do, so I decided one day to sit down in front of it and start typing. I decided to be creative and wrote a cutsie little poem about how the lady says the computer doesn’t work, but it works fine, and then I repeated a refrain that said something about defective end users, etc.
Appearantly when I closed Word I wasn’t paying attention and just hit enter a couple of times and saved this document, which she found when she got the computer home. Luckily I was at lunch when she came in, and one of my co-workers caught me in the back before I went up front and told me to leave for awhile. I guess she yelled at my boss for a good half hour. When I came back he showed me what she had found, she printed it up and brought it in, and he told me, “Never do that again!”

Dude that was TOO funny!!

In reply to DEU (Deffective End User)

There are so many times I wish I could have done something like that. Defective End User. Trouble between Chair and Monitor. RTFM.

User take affront at friendly suggestion

In reply to DEU (Deffective End User)

The biggest mistake I made was to suggest to someone to use Explorer to locate her restored files that I had put in a subdirectory. This user believed that files on a server could accessed by anyone. One day I had to update the OS on her computer and saved her files onto the server, and restored them to a different subdirectory and told her where they were.
She was so upset about my suggestion that she had the director write a letter to my boss complaining about my people skills that I was transferred to another office after that.
I learnt to handle users better after that.

I had a line in some code

In reply to DEU (Deffective End User)

I wrote for a special application. It was meant to be part of a troubleshooting section, the line was rude (and during the writing only echoed to me) but I never took it out in the last version– nothing ever pointed to it, so it would never come up. However, we got a new boss who took it to corporate HQ, sat down to “test” it with one of the HQ techies (neither knew what the program did since it was a specialty application my old boss had me write for a customer) and they never let it get all the way through. But she DID go to the trouble of stopping it and reading the code, caught that one line and used that as “another” reason I should not work there anymore. Eventually, she got her way, but the following Valentine’s Day the owner fired her– she had been there less than a year and had driven off all but one of the people left in her area (and she was working on getting rid of him, too). In the process, she also drove off a few much needed customers.

Nuked entire medical practice server

In reply to What’s the worst mistake you ever made as an IT pro?

After adding a secondary drive to our server, I thought I would be brilliant and I made the secondary drive “Active”. WHOOPS! I lost everything on the server. Our office is a doctors office and we access charts through the server. Luckily the doctors could still get to their patient charts via the internet but it took me a couple of weeks to get everything up and running again. Not only am I the only IT employee for this practice, I run the transcription department as well so I was pulled in so many directions. I thought for sure I was going to be shredded by the doctors but I actully was told, “everybody makes mistakes”. Glad I work with great people. People actually complimented me on screwing up because they said that things work so much better now and for not shirking responsibility. How could I anyway? I’m the only IT person in the building. Anyway that wouldn’t be cool to blame anyone else. I biffed pure and simple.

Did I mention that there was absolutley nothing backed up? I now back-up everything religiously.

Now I am setting up a VPN. Hopefully it wont be another one of the above “learning experiences”. Wish me luck.

Trusting a Remote IT Staff

In reply to What’s the worst mistake you ever made as an IT pro?

The worst blunder was trusting a remote IT staff to guide me through adding a hard drive to a Banyan server. I was working as a field computer technician for a service company and was sent to one of our customer’s sites to upgrade the server. The server had 2 SCSI hard drives and a backup tape unit, the hard drives were address 1 and 6 and the tape was 3. I was told that the company needed more drive space and to install a third drive. I spoke to their IT and told them how the drives were configured and that I did not know which order Banyan used the hard drives.

My philosophy in adding hardware is to make as few changes as possible and I chose to make the new drive address 4. I told the guy who was walking me through the upgrade and he told me that this configuration should work. HE had me format the new drive through one of Banyan’s utilities figuring that this would speed things up. Basically, I was his eyes and hands on site and did what he told me to do.

After the drive was formatted I started the server and let it start up. It seemed to start correctly but some files were missing. The remote IT staff took over at that point and thanked me for installing the drive. They were thinking at that point that the server was working good but needed to move files around to accomodate the new hard drive.

The IT staff finally got the server running again but most of the data was lost. The people in the local office were rather hostile towards me for losing the data and causing their server to be down for longer than planned. They did accept that the faulty backup was their problem but would not believe that I was following the instructions by their own IT.

Hey, if the guys who walked me through the setup are reading this, I am sorry I was not more adamant about verifying the setup. That would have saved all of us the hours of grief.
Two hours later I was told to return onsite. It turns out that the drive I was told to format was actually the drive that the data was stored on. I was sent back to restore data from their back up tapes. That is when we discovered problem number 2, none of the backups were done correctly.

Two: One technical, one not

In reply to What’s the worst mistake you ever made as an IT pro?

Technical: Many years ago I had to insert a hardware performance monitoring probe into a live mainframe computer. I put the prob in the wrong place and brought the entire system down.

Non-techical: While I was presenting to the IT Leadership staff, which included my boss, I told him, he was completly wrong about something, which did not make him look good in front of his boss. He did not like that very much at all and pretty much ended my career at that company and eventually was laid-off causing an abundance of grief for me and my family for two years. I’m still paying for it still today.

Gotta love backup tapes

In reply to What’s the worst mistake you ever made as an IT pro?

Having sys admin responsibility for a number of UNIX boxes (in the days of the PDP-11), I was interrupted several times while doing cleanup work on various system directories.
After the last interruption, I did an “rm *” without verifying the directory I was in. As Murphy would have it, I was in /dev. UNIX doesn’t take kindly to losing that bit of information 😉
Fortunately for me, The 2AM backup tape was on the rack – and was good. All was back to normal in about 20 minutes. As we were a beta test site for some company software, the users just assumed it was another glitch in the “new stuff”. I kept my mouth shut and the computer room techs wouldn’t say anything negative about the only sys admin who didn’t yell at them when they phoned for help at 3AM. It pays to be nice to other people, even in the middle of the night 😉

Restored to original destination

In reply to Gotta love backup tapes

Repeat after me, never restore to original location. Navigate to the location and select and this will prevent you from performing what I am about to describe. while trying to setup a test environment, the network admin using the weekend backup was going to restore to the dev box to emulate the production environment for testing, but instead forgot and overwrote half of the production files with the weekend files causing complete shutdown of all systems for a distributor/manufacturer for 1 complete production day. Imagine the cost involved of lost production and shipping time as well as all the time required to re-enter all of the accounting information that was lost not to mention the time involved in remediation of records that were in limbo. So, remember when restoring, manually select the destination as it will always remind you of where the data is going.

Take down a stock trader

In reply to What’s the worst mistake you ever made as an IT pro?

I’m a telecommunications guy and was a branch engineer for a company that sells digital phone systems. We had a customer who was a stock trader with about 50 to 60 employees call us with a strange problem with their phone system. I went in remotely via the system modem and checked a few things out and determined that a system reset would clear the problem. Unfortunatly at the end of the day when the stock exchange closes all the traders are making phone calls so the system was full of calls. We have a mechanism in the system that can busy out all the lines and as the users complete their calls it busies out the line. I watched the status screen until the last user dropped off and then performed a reset. I forgot that resetting the system also drops the remote modem connection so I was knocked out and now the system came back up without any lines working! I could not even call them to have them put me back in the system. I had to fax the user a message that a technician was on the way before they open in the morning! Luckily the tech arrived before they officially closed and all was well. Mind you I already had 6 years of experience with this system and just had a mind blank that day.

What’s That Red Light Mean?!?!

In reply to What’s the worst mistake you ever made as an IT pro?

Well, how about taking down the entire Server room: Unix, Mainframe, and Windows!!! We had just installed a new behemoth UPS. “Wow!” What a wonder it was. Now we could feel confident that in the event of a power failure, we’d have a whole 25 minutes to arrive on site and down everything properly to prevent data loss.

But what would happen if the UPS went down? No one may have been asking themselves this question, but we all got to find out anyways, courtesy of yours truly.

On one of my routine morning checks of the servers, following the installation of the UPS, I noticed a big red light on it.

Well, in my experience, a red light means that something’s wrong. I stepped closer to the unit, and reached out my hand to touch the light ever so slightly… WRONG ANSWER!!!!!! It wasn’t just a light, it was a button. And it had these three letters on it: EPO For those of you unfamiliar with this acronym, as I was, it stands for “Emergency Power Off”. And that’s just what happened. Power for the entire room was immediately terminated. I gulped hard, quoted Butch Cassidy as he’s jumping off of the cliff, and then called my manager. “Uh, I think we need some help down here. I just killed the power in the server room.”

Well, believe it or not I did NOT get fired (even though if I were my manager I would have drop-kicked me right out the door!).

After that the Facilities department made a little plexiglas cover for the button save others from the terrible fate if they were also ever inclined to press the BIG RED LIGHT!!!

I deleted my professor’s account

In reply to What’s the worst mistake you ever made as an IT pro?

I work at a small private university and was also attending classes at the time to finish my degree. I got my first ID to delete and popped right in and deleted it – only it was the wrong one. The ID I deleted was that of one of my professors that semester for a programming class. I did away with all his files and emails – and the man was working on his Ph.D.! I nearly died! We were able to recover his files, but not his emails. He was exceedingly gracious about the situation. And no, I didn’t fail his class. In fact, I got an A!

NT driver

In reply to What’s the worst mistake you ever made as an IT pro?

Back on 1998.
Load a printer driver on NT 4.0 was the challenge specially on the production prinetr server ooppps.
The problem was It was a sunday an I thoght i was alone on the firm.
(I forgot that all the lawyers from the firm were in a meeting on the conference room)
Needless to say Lawyers like to print a lot.

Remember the VAX

In reply to What’s the worst mistake you ever made as an IT pro?

I used to manage several VAX 11/780’s These had a wire-wrapped back plane that was custom created for each system. On a system upgrade one of my peers from another state had to put another wire on the backplane for a “jumper” due to a new hardware upgrade. I let him.
He put the wire on the wrong post and fried the backplane, melting about a dozen wires. Frying several of the cards in the system. I had to re-wire wrap the backplane and coax the system back to life! Needless to say – I was lucky to keep my job. (Don’t get me started about the other guy!)

The first Netware 4 install I ever did

In reply to What’s the worst mistake you ever made as an IT pro?

went so well that IT Admin and I (an old friend) went over the pub – luckily, I had just one beer – before returning to finish the job.

To make some minor amendments without mounting all of the disks, I brought the server up “-NA” which, in earlier NetWare versions prompted for a server name, which could be anything as it’s lost on reboot.

NW4 does the same – nearly!.

The IT Admin was p:ssed and was annoying me so when the “Enter Server Name” prompt came up I type FUCK_OFF_YOU_BASTARD to give him a laugh, edited the config and restarted the server. It all tested OK, came up “MERIDIAN_01” and all of the IT container were able to log in. Alas, we were all in the Server Context…

“OK, everyone. You can log in now”

Login: Mavisp
Cannot find server object FUCK_OFF_YOU_BASTARD
F:>

I’m in London, so after troubleshooting for a while (after I’ve hidden for an hour, calmed down and overdosed on tranks), I called Novell Tech in Germany but he couldn’t seem to grasp what was going on. I called Provo and told my sorry tale and I thought that the Tech was going to explode. When he’d told everyone in the building and they’d all had a good laugh, he gave me the fix to remove ghost objects.

I asked the Tech that if I’d have typed PISS_OFF_YOU_BASTARD, would anyone ever have known as the server names are checked alphabetically. I’d have been safe.

Moral. always expect the unexpected.

What a difference a letter makes.

In reply to What’s the worst mistake you ever made as an IT pro?

Typed “tar -cv” instead of “tar -tv”.
The memory is painful.
Here’s my “stupid user” story of the week.
Cheers!

Did I do that?!?

In reply to What’s the worst mistake you ever made as an IT pro?

We had received our new APC Symmetra UPS within the past 1-2 months. I was configuring the APC PowerChute Plus software on our servers so the servers would be shutdown automatically when the UPS switched to DC for longer than fifteen minutes.

On the configuration page for the client software was the [checked by default] option “Turn off the UPS after the shutdown finishes.” I didn’t read the documentation and did not catch this. When I tested the server configuration, by manually putting the UPS on battery, the server shutdown as expected. Then, the UPS shutdown also … taking our entire server room (servers, core network equipment, telephone system, etc.) with it! Needless to say, I felt pretty stoopid!

🙂

Added TrialWare to App Bases Classes

In reply to What’s the worst mistake you ever made as an IT pro?

I changed the application base classes used in a range of different products, and while I checked the building etc. I forgot to check the operation of the various applications and as a result, brought 5 product lines to a complete halt.
After 1/2 day of running around I was given a roundly sarcastic talking to and then told how to fix the situation. After 8 months on the job it was an eye opening experience so that I have slowed down everything I do.
So now I get complaints about how long everything I do takes.
Damned if I am not careful, Damned if I am careful

This was not entirely my fault but…

In reply to What’s the worst mistake you ever made as an IT pro?

I had just started a new job. I was tasked to port this huge database and deploy an isolated test copy on a private box so I could beat up on “realistic” data. There was the master and the slave. Obviously you dump the slave, because a database throws up a write lock while it’s dumping and you don’t want to dump a live database in the middle of the day with several hundred users connected to it. You can dump the slave, because write throughs to it will just be deferred until you’re done.

Did I mention this was a HUGE database with several million records which took more than a trivial amount of time to dump itself especially if you’re piping it directly to the network. 😛 Well apparently the site admin had been a little inconsistent with naming the servers, and he had very strong opinions on what their names should be, which conflicted with what the boss actually named them. As you know if you run linux, server names can come from one of several places and at that time there was nothing to assure that the names were consistent. Being brand new, I believed the server names I was told and didn’t check against IP numbers, and inadvertently started dumping the master at 10:00 in the morning. Boy did my phone ever ring!
I had just disconnected about 600 remote users in every state in the USA and a few foreign countries.

There’s Actually Two Big Goof-ups

In reply to What’s the worst mistake you ever made as an IT pro?

The first was when I start “officially” as a member if the network administrative staff. I accidently moved the complete email user list into another folder area. Took a while to figure out what had happened and some more time to “fix” the mistake.

Second time occurred when I installed my very first “high-speed” CPU onto an existing motherboard. Thought I’d save the client some money and use an existing CPU cooling fan. Wrong! It took about 15 seconds to fry the CPU. An expensive mistake but also a learning experience.

Three Characters

In reply to What’s the worst mistake you ever made as an IT pro?

Some years ago, the company I was working for got a new client. One of our tasks was revamping the data structures to align with the business needs. One of these needs was to print names and addresses for large volumes of mailings. The name field had been defined as 85 characters and we decided to truncate it at 33 after careful inspection. We started a process to run over night to change the fields and structures.

The next day we got a call that all the names were now three characters long. No problem, we had backup tapes going back for over a year. When we tried to reinstall yesterday’s database, we discovered that the backup process was only saving the operating system – no data.

You can imagine the nightmare it took to reconstruct the database while our client was essentially out of business for over a week. Dropping one character (in combination with other errors) cost millions.

Well,

In reply to What’s the worst mistake you ever made as an IT pro?

I once thought I was mistaken about something but it turned out I wasn’t 🙂

Seriously two stand out. I once rolled out a (THE, at the time) server to get the numbers off the back, and miscalculated the length of the power cable…

I have also dropped a brand new 21 inch monitor. Very loud!

choosing IT as a career w/o BS degree

In reply to What’s the worst mistake you ever made as an IT pro?

The single biggest mistake that I made was starting an IT career by going to a two year tech college and getting an associate’s degree whose credits won’t transfer to a four year school. (Of course, I made the mistake of asking them if their credits transfered instead of asking the school that I was going to transfer to.) The first 30 years I progressed from assembler/cobol/dbase/foxpro programmer, to a windows administrator/IT mgr. However, the lack of a BS finally caught up. My last employer went bankrupt, and now employers won’t consider me because my resume doesn’t list a BS degree along with a boatload of certs from Citrix, Cisco, Msoft and Redhat. I’m in a deep hole in which it seems there’s no way out!

Reply To: What’s the worst mistake you ever made as an IT pro?

In reply to What’s the worst mistake you ever made as an IT pro?

I was trying to clean out a cache directory on an old IRIX server we were running a proxy, website, and email server on…. instead of “rm -f ./*/*/*/*” or whatever the command was at the time, I forgot the . and noticed alot more stuff than usual ripping by on the screen. Rookie mistake…. no excuse. Shameful….

Re: What’s the worst mistake you ever made as an IT pro?

In reply to What’s the worst mistake you ever made as an IT pro?

The worst thing I ever did came as a result of instructions given to me over speakerphone at the place I was consulting at by the technical support staff of a proprietary software application. My client was having trouble with their application not loading customer data. I came into their office, called the vendor and after they talked me through a few things, had me navigate to the installation folder of said application. I was then instructed to read a list of files in this folder. The guy said oh, delete this file and gave me the filename. About half a second after I deleted it (I am a power user who holds down the shift key when he deletes) he said oh wait?not that file. By then the damage was done.

About them backups…

In reply to What’s the worst mistake you ever made as an IT pro?

Probably my worst mistake was when I setup a few backups jobs. A program we use at work normally makes a backup file of all its transactions just in case something were to go wrong. The manual itself stated to backup these files instead of the main data files. Not to mention, the backup files are 1/10th the size of the regular files – which set of files do ya think I’m gonna backup?

Well anyways, I got a call saying that a database got corrupted. So I go to the backup tape, copy the backup file, and then revive the database. I give a go ahead to try it out.

Well, I get a call a few minutes later stating the file is 4 months old… and is not the recent revision! I double check the tape, and I had backed up the backup files correctly…

So I make a few inquiries and it turned out my boss decided to shut off the auto-backup feature of this software… and since I had backed up only the backup version of the database… well, let’s just say my boss wasn’t too happy.

Lesson learned: backup everything, and not just what the manual says…

Things I have learned so far from this thread

In reply to What’s the worst mistake you ever made as an IT pro?

(Ok, I knew them already, but just in case):

1. Make backups daily.

2. Don’t touch the Red Button.

3. Make backups daily.

4. Make sure the backups WORK.

5. Spell check your commands, before you hit the “enter” key.

6. Make backups …daily.

7. DON’T EVER TOUCH THE RED BUTTON.

Don’t forget…..

In reply to Things I have learned so far from this thread

Do not mess with the 110/240 volt switch on the back of the power supply – or check it’s all the way to the side it’s supposed to be. They tend to go bang, and emit ‘magic smoke’ otherwise, having seen this myself. The phrase I used was ‘wow! that can’t be good.’

More to learn

In reply to Things I have learned so far from this thread

Pay attention and give the benefit of doubt to end users.
Some time ago I was very tired of a long day of users phone calls (working on a help desk)(I think it’s hell’s closest area) and I got a call from this user that can’t login.
I asked him what was his username and he answered me “ogemel” and I said, “I’m sorry, that should be your password, you shouldn’t reveal it to no one, please tell me your username, it should be similar to your name, what’s your name, please” and he answered again “ogemel”, then I gone beserk and asked “Can you hear me? I dont need to know your password, tell me your name, where are you calling from?” and then my coleague at my side, called me, “Hey, be cool, we have a user called Holger Melo, and he has a funny accent”.
My face turned red and I had to disgise my stupid attitude with all education I could find.

mistake

In reply to What’s the worst mistake you ever made as an IT pro?

was plugging in a new server and apparently was too much power for that rack.brough down 10 servers.mostly web servers.but it also had our main dc and exchange….yea that produced some calls.my manager found it funny.?.?

put it wrong date for a job and made the system charge everyone single account an overdraft fee.fix…..manually reverse all 30,000 accounts or pay 25,000 to the company that provides the server and software to fix it.didnt get in trouble for that either.

Opened my mouth

In reply to What’s the worst mistake you ever made as an IT pro?

and inserted my foot to the hip.

While working for the military, had a subordinate who would take off for various reasons, frequently not leaving notice of where he was going or when he was coming back.

His wife had a habit of frequently calling him at work.

She caught me on the phone at the wrong time and I just blurted out in frustration that ,”I don’t know where the hell he’s f*cking off at right now.”

Manager damn near canned me on that one.

Wrong window

In reply to Opened my mouth

Had been working an intermittent issue with a remote branch’s T1 – it was flapping all day long, so I had telnetted to the thing to look at interface stats and kept it open and was checking every once in a while to see if it was up or down.

While I waited, I was consoled into another router on my desk, configuring it to go to another branch that was opening in a couple weeks.

Well, when you have 2 xterm windows open it’s pretty easy to type into the wrong one. I assigned future IP for the router on my desk to the one at the remote office. Bye-bye remote office. Fortunately they considered it a part of the intermittent issue and I called and told them I was doing some troubleshooting with SBC and asked the local (non-technical) staff to power cycle the router – nothing was saved to NVRAM so all came back up and I told no one of my stupidity.

What’s the worst mistake I’ve ever made as an IT pro?

In reply to What’s the worst mistake you ever made as an IT pro?

Becoming one.

Seriously.

I work for a company that has no concept of the NEED for training or the need for the proper tools.

Lack of management support and vision stops me from being able to do my job.

I’ve been in the same place for 11 years, doing the same job. Increased responsibility (with no authority, I might add) as well as tasks, with nowhere near the same levels reflected in promotions/raises. Management second guesses me because they read something in Newsweek and thinks it applies to us.

I’m also tired of being the sole Admin around here. I have a pager, a CrackBerry and another cell phone. I work nights, weekends, vacations…

Problem is, I make enough now that leaving would mean a pay cut – at least for a bit. That makes it difficult. And since I don’t have a Bachelors, that ties me down even more.

So I’m working on some certs. Certs are BS, I know: it’s a test that gives you a piece of paper. Cert mills have ruined their value. But if you don’t have one, you don’t even get looked at.

Hopefully, in a year, I’ll be working somewhere else for a more reasonable employer.

If I had it to do all over again, I’d have chosen a different field. Possibly not even computer related (“talent” is getting cheaper and cheaper, so why work with computers?). Or, perhaps, my employer is the issue. Hard to tell, as this is the only purely IT job I’ve ever had.

Anyway, you asked. I (kinda) answered…

You’re not alone!

In reply to What’s the worst mistake I’ve ever made as an IT pro?

I totally agree, because I’ve been in the same environment, but now I’m unemployed. I have 30 years with a variety of exerience, but only in one man shops, with no BS degree and only a couple of certs. Without a BS degree, recruiters and employers don’t even look at my resume. Best of luck to you!

Bumped the key – OOPS

In reply to What’s the worst mistake you ever made as an IT pro?

Was working on major changes to a customer’s inventory structure, and had a long SQL UPDATE to add leading digits to their part number. I got as far as “UPDATE INVENTORY SET PART-NUMBER=” and bumped the key with my notepad that contained the rest of the statement I was typing in. Since it was a server-side request, I couldn’t stop it as it went through the db wiping out all of the customer’s inventory.
THANK HEAVEN I had sense enough to have made a full backup and lock the db exclusive BEFORE I bumped the key. It took 10 minutes to restore, but I was sure glad I had the Backup!

Pride is the greatest fault…

In reply to What’s the worst mistake you ever made as an IT pro?

As a small business, IT consultant, I was perturbed when I could not get the WiFi working on a Gateway notebook…Immediately, hardware was the cause..according to me. Finally, due in part to my poor wife drawing her line in the sand (based on my rantings), I took the machine to a certain BestBuy affiliated competitive group…$50.00 and 5 minutes later, the youngster behind the counter showed me where the WiFi button was located on the keyboard…live and learn…

Vacuum in networking…

In reply to What’s the worst mistake you ever made as an IT pro?

This still haunts me to this day. Every year we have a main sales day where we put all of our stock (I work in the vehicle industry before you ask) on special and we have a massive fuss over the whole deal.

Anyway, I was in my first year and was just running around making sure everything was working and in order. Only one department was using their computers, so I was asked to keep an eye on the rest of the “technical” stuff. I noticed that the pathway between our comms room and the lounge (or lunch) room was covered in limestone footprints and looked pretty awful. Being the conciderate type I decided to grab the vacuum cleaner and clean it. Being that there was no power outlets near the corridor, I got an extension cord and plugged it into a spare outlet on the comms room wall. Little did I know that all the outlets ran off the same curcuit and sure enough, the trusty vacuum cleaner overloaded the circuit and threw everything on it out!!

The UPS’s got locked in a power overload state and refused to power on – the main circuit breaker blew and I had siumltaneoulsy knocked out all of the phone system and computer network in one hit!

Half an hour later and absolutely soaked in persperation I managed to get everything back online. Now whenever there is a network slowdown a manager always makes the comment “Not using the bloody vaccuum again are you??”

Gotta love that 🙂

Is that UPS shut off?

In reply to What’s the worst mistake you ever made as an IT pro?

My company has been building a new datacenter for a trading company in NYC. We recently implemented an IBM blade center. So my boss was working behind the rack moving wires and cleaning up. He must have kicked the power on the UPS and became nervous that something shutdown. I am working in the front unaware of this as he calls out, “is the UPS shut off”. I take a look and its on. I tell him no and shut it off for him. He yells, I didn’t want it shut off. Oh crap I managed to sever the SAN from the Blade Center in one click. Luckily after rebooting all, the attached storage came back online and since it was 5:30 on a Friday night, I lucked out… nobody noticed.

Everyone please read, 4 year blunder w/o answer

In reply to What’s the worst mistake you ever made as an IT pro?

I have been a consultant, designer, implementer of NT/AD W2K based IP networks for several years now. Several years back I lost 2 different person’s PST files with still no clear clue as to how and why they couldn’t be recovered. It happened sometime after I inherited the support of a small office running SBS 2000 and eventualy led to the conversation of yes we had Exchange email but it hasn’t worked in a few years. Upon request, I installed, configured and pushed out Norton’s Enterprise Antivirus client to all computers but a few reported a strange error. The error had to do with Norton’s autoprotect not being able to contact the Exchange mailboxes using Outlook 2000. FAQ from symantec and logic suggested I remove the failed email accounts from Outlook and leave the Internet mail (pop)accounts configured. It sure sounded fine at the time. It wasn’t until a second person reported the problem and I removed the Exchange account that everything disappeared with no traces of a PST, OST, etc. that I began to realize I was in trouble. I looked high and low, on the net and support.microsoft.com, file recovery programs, but to no avail. What occurred is the smtp Exchange accounts were created first and sometime later, I guess after they failed, pop internet accounts were setup and defaulted to using the same PST files for storage. When I removed the Exchange accounts it also permanently removed the PST files associated with them regardless of other mail services still using them. I swear to this day this was my hardest lesson ever learned as I got stiffed on the monthly invoice for all billable hours and there is no information on the net that Outlook 2k would do this. I am still curious if anyone has any further insite on this. Sorry for rambling.

I Broke Printer’s Motherboard

In reply to Everyone please read, 4 year blunder w/o answer

About 5 years ago someone gave me a motherboard of a printer to troubleshoot some minor issue. I put that motherboard in a shopping bag and hanged that bag on my bike’s handle. On my way back to my office, when I took a sharp turn, that motherboard got stuck between bike?s handle and oil tank and broke into two pieces 🙁

I spent whole night in joining the broken electrical contacts but that board still did not work and I had to pay the cost to its owner.

The short invoice

In reply to Everyone please read, 4 year blunder w/o answer

I used to work for a small company whose customers used to short invoive (actually, short the payment) to get the owner’s attention so he would call and actually talk to them about some problem they were having.

My mistake? He asked what he should do about “this short invoice” and I, fresh from a stint at a collections law firm, suggested he call them and see if they could collect the invoice … while the customer was on speakerphone.

On the plus side, he never asked me that stupid question ever again, and at least one customer never shorted an invoice.

Working for a SCO UNIX shop

In reply to What’s the worst mistake you ever made as an IT pro?

Let myself get hired by this place because their customers were buying PC’s instead of expensive terminals for their UNIX servers, and they needed a windows guy.

…

Dumped the USAF simulator system

In reply to What’s the worst mistake you ever made as an IT pro?

My buddy and I were the ‘computer experts’ as collateral duties at Navigation school. Big extensive systems provided by the USAF for flight navigation training. Big RED button on the panel, covered by a clear, plastic flip-up cover. EMERGENCY. That sort of thing. We ran a LAN, had worked on computers since the 286 and earlier. What does Ed do?

Pop the cover. Push the button. Guess what? Everything shuts down very quickly and lots of civilian technicians start setting their hair on fire.

Lesson learned? Don’t let Marines near your Air Force computer simulator ’cause we’re gonna push every button…

Deleted all permissions.

In reply to What’s the worst mistake you ever made as an IT pro?

We had this file server that serviced over 1000 users. Well, I was trying to fix one users permissions and by accident deleted all permissions to a few of the most important files on the server.

Luckily I had some contacts that I new and I called them, let them now what I had did. After a quick laughing session they quickly educated me on my mistakes and how to fix them. Also to my advantage this was during the graveyard shift. A few managers were a little pertubed but thankfully I resolved the issue before the big “cheese” showed up. But I can say that I gained a ton of knowledge about permissions.

Sometimes I find that I have to make mistakes in order to learn. How about you?

I still do it!

In reply to What’s the worst mistake you ever made as an IT pro?

I’ve assembled and installed computers for about 12 years. I can install any type of computerequipment for anyone, except for myself. And I can’t find the problem no matter how hard I try. Then I have to get one of my colleagues to look at it and endure the smirks. On my first computer it was a screw shortcircuiting the mainboard to the chassis, on the next the processor heatsink touched a condensator under the heatsink, on the third I left the ClearCmos jumper in place, on the fourth I had left both harddisks as master, even after checking twice. And so forth… Strangely enough, none of the computers got broke and worked perfectly afterwards. Is there some kind of Murphy’s law for working on Your own computer?

Registry Deletion

In reply to What’s the worst mistake you ever made as an IT pro?

I was working on a spyware-infested computer that simply would not stay clean. So I’m going through the registry and come across a key called “Search Assistant”. Well it sounded like spyware, so I deleted the key. Apparently the key was an integral part of Internet Explorer, because it couldn’t search to any web pages after I deleted the key. I ended up having to reformat and reload. Lesson learned.

Oops! No more Alumni

In reply to What’s the worst mistake you ever made as an IT pro?

Back in the early 80’s I was scrubbing the alumni database for a major university for duplicate records. Due to a typo I managed to deleted almost all of the 40,000 records. I called the data center and said something like “um, there’s a problem with the file you sent me. Could you do another export?”.

Oops, I’m sorry.

In reply to What’s the worst mistake you ever made as an IT pro?

Years ago I was working as a system integrator (read outside tech. consultant) and was contracted to an insurance agency to roll out a new e-mail system for their office. I had a good working relationship with the office and was well trusted by their management folk.
I arrived mid-morning with the required equipment and their Accountant /?CIO? told me where to make some room for the new server case and then left me to work my magic. The new mail server was fairly well pre-configured so all I really needed to do was “nudge” one of the older servers over a little, connect the new mail server appropriately, tweak some final settings in the mail software and I was off to the desktops.
The old server was heavy and needed a little wrestling to slide across the floor and just as I reached around the back to get a good grip I heard the sound level in the room change. I had just turned off their Agency Management database server while a couple dozen people were logged in and working.
I had no choice but to sheepishly seek out the woman who had just entrusted me with her server room and explain why her phone was about to start ringing.
The database server was well-configured and very little data was actually ?lost? but I had the email system up and running for hours before the mirrored set had finished its repair process.

Domain Computer

In reply to What’s the worst mistake you ever made as an IT pro?

It was on the early days of my first job.
I removed the computer from the windows 2003 domain and I do not know the local user id/password information. That computer had lot of valuable information and I was really panicked.
Finally I found some online tools which could log on to the computer without a user/password which saved me.

Sheesh

In reply to What’s the worst mistake you ever made as an IT pro?

Maybe I shouldn’t fess up to these but here goes:

1. While in college I interviewed for a PC Technician position. The IT Director knew I was going for a IS degree and asked what my least favorite programming language was. I told him I hated COBOL. Naturally I found out, after opening my mouth, that they were a COBOL shop. But I still got the job and upon graduation was promoted to programmer (I’ve since moved them from COBOL to VB.net and C#).

2. Early in my career I accidentally disconnected the external SCSI backup drive for a Novell file server. Without thinking I immediately reconnected it (while it was still plugged in to the wall). Fried the server’s MB and the tape drive.

Router Bug Knocked Biz Offline for 3 Days

In reply to What’s the worst mistake you ever made as an IT pro?

In 1994 I was finishing up a campus network for a very large company, the last piece being the core router that would tie all the campus buildings together. Unfortunately, the mortgage company was on a Token-Ring network, and my testing plan included only Ethernet(no T-R in the lab), so when we upgraded the router a bug in the T-R framing size software knocked their SNA Gateway offline, which we didn’t discovery from Friday night to Monday. After three days of troubleshooting the wrong thing (gateway) we discovered that the ‘upgraded’ microcode in the router had unmasked a configuration error. In that time the business failed to complete financing on almost $4 Million in paper. I went through a formal loss review and what saved my job (with only a verbal reprimand) was that I had tested the config with available equipment and specified in my migration plan specifically not to touch the T-R configuration. Lesson: Test Everything, and specify what to do, what not to do – and add a backout plan (e.g “revert to last stable configuration if new config is not stabilized by 2AM)”.

Unintentionally testing the UPS

In reply to What’s the worst mistake you ever made as an IT pro?

I was still pretty green in my 1-man IT position for this semi-small company of around 80 employees. We had a new AS400 installed 6 months earlier and it was purring like a kitten. Well, I got a call from some sales pirate wanting to sell us components for our AS400, and wanted the model and serial number off it so that he could target stuff to us. I should’ve said “Go away” but no, I felt obligated to get this little bit of info for him (I am much wiser now.) I went into the computer room and got behind the AS400 and was sort of squatting or sitting on my haunches trying to find a name plate or something. I lost my balance and fell backwards. I tried to catch my fall (didn’t want to land on anything crucial) and reached back ato brace myself against the UPS. If any of you have ever seen Lord of the Rings (and I’m sure many of you have) theres a scene where Frodo falls and the ring mysteriously falls right onto his finger. That;s what happened to my finger. The only thing that I caught was my index finger onto the UPS power switch. I heard hard drives and fans wind down to silence, and sat there going, “Oh crap.” I switched the power button back on, and thought “Maybe no one will notice.” Just as Frodo disappeared and caused quite a ruckus, everyone’s session disappeared. It took all but ten seconds for my name to be the most popular one on the paging system. It took well over two hours to reboot the server and fix all of the processes that it was in the middle of during the “incident at the Prancing Pony”.

Hardware Pirates Beware

In reply to Unintentionally testing the UPS

Oh – and hardware pirates get no mercy from me anymore.

UPS failure

In reply to Unintentionally testing the UPS

Your story sent me back to one of the worst times on my job…

The UPS was loaded with 8 switches and around ten servers (Big mistake). It was going like that for almost five years. One sunny day, the UPS gave me a warning that the battery test failed. I thought that it still going to work for a few weeks (Another big mistake) and I was going to look into it at a later time. Couple of hours later, the UPS shuts down and with it all my devices. My luck was that this happened on a Friday at 4 p.m. and it didn’t impact the production very much. It took me a whole weekend
to make the network functional again.

Me too

In reply to What’s the worst mistake you ever made as an IT pro?

After 15 years of experience in IT someone come to me with an old 586 machine.
After disassembling and cleaning (I never see a computer with so much dust inside) I begin to assembling back all the components. I forget that this type of CPU you can plug in even in wrong position, so I did it. After power on of course a lot of smoke and one burned CPU.

Returning PCs

In reply to What’s the worst mistake you ever made as an IT pro?

My biggest excuse of a mistake was sending back 3 brand new computers because I couldn’t turn them off. It seems that a 5 second timer was added to powering off (you know, you have to hold the button in for 5 sec) and somehow in all my reading I had missed that new feature.

Three new PCs and one vendor’s course on BIOS settings and everything was fine.

Bout that time, they also changed…

In reply to Returning PCs

the way that the power worked. They went from “power button kills all power” to “system board still live with power off” mode…. and it was time to upgrade ram on the production web site…

I am *now* the ONE you want to upgrade ram… After frying the dual processor mobo, I was able to locate one in California and have it FedExed in the next day.

Till then, the Web Site was down for the firm I work for… and I now unplug every PC I work on before opening it up. I am also religious about the 3M static wrist band…

Older and wiser I am…

Major Oops.

In reply to What’s the worst mistake you ever made as an IT pro?

A long while ago, I was an application developer on a honeywell mainframe system. While developing an application I had to erase all the records from the master detail file (I thought I was in the test account), will I was in the production account and I wiped out the complete history of all work for the company for the last 8 years. I immediatly walked into my bosses office and said “Jim… I thinked I fu**ed up” We had to restore all the tables from the previous nights backup and then everyone in the company had to redo all their work for the day. This was the days of mainframe and flat files, no databases and no archivelogs. To this day, I am still parionid about what schema or directory I am in before I perform any changes. I might make mistakes, but at least I never make the same ones twice.

Similar but Different

In reply to Major Oops.

I once installed an early beta of an application I was developing on an Insurance companies network. In the early days of Windows 3.1, installing a new app could break a machine. I installed the app, rebooted and nothing… Shortly after this, the secretary came to me and asked why she couldn’t print. Thinking on my feet, I told her that I was updating the network and she would be able to print shortly, and by the way, could she give me the Windows disks. After much swearing, I reinstalled Windows and everything started working just fine. The lesson learned, always use a professional install application!!!

Upgrading Home PC

In reply to What’s the worst mistake you ever made as an IT pro?

Windows 3.1 was perfectly fine. Why, oh why, did I ever look at the shiny blue box with the 4 color logo? Like a lemming, I followed the trail right to Bill’s pockets, inserted enough money to feed me for a month in college, and hurredly made the trip back to my dorm room. I quickly whipped out and installed… junk.

Luckily, I still had my 6 Windows 3.1 disks, 3 MS-DOS 6.22 disks and ran a FORMAT c: on my 60MB harddrive, edited my autoexec.bat and config.sys files to optimize the 4MB of RAM and continued to play “Scortched Earth” with my room mate.

Public Folders

In reply to What’s the worst mistake you ever made as an IT pro?

I once accidentally deleted a public folder on our Exchange Server which held over 1200 contacts – no backup.

Luckily, I was able to extract the data from an old .edb file.

Assuming standardized installations existed in remote sites

In reply to What’s the worst mistake you ever made as an IT pro?

I was once working for a large package transportation firm where we had to rollout a memory optimization procedure to increase performance due to the escalating size of some of our executables. We (I) mistakenly made the assumption that this same procedure would work in all the remote district sites because the consensus held that the computers in question were all set up the same way. In fact, this could not have been further from the truth, as many had installed personal software which had modified the config.sys and autoexec.bat files on these machines (back in the DOS days). BIG mistake. The net effect: this rollout worked for only around 30% of the systems involved, which resulted in substantial time spent on the phone with the users reworking their systems to get them back to operational status. DOH!

spliting tea on the GM labtop ;)

In reply to What’s the worst mistake you ever made as an IT pro?

It was my first day in that company, GM has just purchased a labtop and ask me to set it up to the domain, then i was drinking tea and then the labtop become a larg flat tea cub, and i was fired 🙂

The Worst Thing Ever

In reply to spliting tea on the GM labtop ;)

My boss was goin’ abroad for few days, so he wants me to setup his personal new sony viao to install application and copy data, it somehow accidently scrached by RJ45 connector when i was tryin’ to close laptop, after that i lied to him that i don’t know how did it happend, he ask me to tell the truth, then i confessed that it was a mistake, i wasn’t fired but i lost my self respect and demotivated by this event i m still working there and lookin’ to switch or leave for abroad that was the worst ever day of my life 17 sep 2005

Biggest mistake?

In reply to What’s the worst mistake you ever made as an IT pro?

Accepting a change in my job status to “Salary – Exempt” with a pay raise. Never really had overtime to speak of before that. But ever since I no longer get paid for it….

My favorite goof was someone else’s. (Naturally.) At the small hospital where I was working, we had put in a new UPS and everything was on it. It was a nice big unit, almost chest height. To power it off, you had to push and hold two buttons about an inch apart, just to make it hard to shut off by accident. Well, our cable guy had to reach some devices that were high up in the computer room, so he stood on the conveniently located UPS. Turned out his foot was just the right size to hold in both buttons at the same time…..

It’s probably good to make these dumb mistakes, so we’re a little more patient with the person who calls us in at 5 in the morning because a crucial printer isn’t woking, and you fix it by … pushing the power cord in a little further in the back of the unit.

Test Networks

In reply to What’s the worst mistake you ever made as an IT pro?

When using a test database to test changes before putting them in a production system, make sure you are actually in the test system itself and not the production database. That will cause some problems.

Filling Storage

In reply to What’s the worst mistake you ever made as an IT pro?

At last we had internal email. I had just figured out how to do MSAccess 1.0 stuff. I wrote a long document, plus screen shots, covering my discoveries and sent it out.

“What are you doing that is filling up our storage?”

“I sent an email to a heap of people, but that is all.”

“How long was the attachment?”

“Why would that matter? Surely Outlook just points everyone to the same copy?”

(I was a poor innocent soul back then.)

“Well, as soon as we free up memory your email’s attachment is filling it!!”

You guessed it, I had sent the email to everyone using MSAccess at our place… we had rather a lot of them.

Can I watch you work for a while?

In reply to What’s the worst mistake you ever made as an IT pro?

Something was troubling the hardware engineer. (Unknown to me, the alarm system in the main control room was spewing out masses of alarms, serious alarms, like “radiation counts souring sky-high”.)

“Can I watch you work for a while? We think you are the common factor in a situation we are analysing.”

I could not see how, but not wishing to be unfriendly I said, “OK, be my guest.”

It was late at night (1968) and I was testing a burst-can detection program in a nuclear power station. The paper tape shot through the reader at 1000 chars per second. It generated heaps of static in the plastic holder. I amused myself by seeing how long the flash of static to my pencil could get. I got about 1 inch that night.

He leapt into the air, fully awake.

“Eureka, that explains all those alarms!!!”

Anyway, a few weeks later the false ceiling caved in. It was due to accumulations of dust from holes in the concrete for rock-bolts to hold services such as heating pipes being added way above the computer room.

The cold air and dust did massive damage. These same engineers then had to replace heaps of relays, and our drum storage (pre-disc days).

I didn’t feel so bad after that!!

OOOPS!!

In reply to What’s the worst mistake you ever made as an IT pro?

When I was a very new network admin, I needed to work on a printer problem. The printer was right next to the server and the server UPS. I was leaning to try to get a better angle on the printer, and heard a “click” down near my foot. I had just powered off the server. I brought things back up, apologized to the users who had lost their work in progress, and then I built a little cardboard protector around the power switch.

People were pretty good about it, although I did get some ribbing for my “big feet”….

Well, there was this one time…

In reply to What’s the worst mistake you ever made as an IT pro?

Once, when the building was burning down, I grabbed a co-worker versus the backups. My boss never did forgive me for not getting those backups…

LOL

In reply to Well, there was this one time…

Well If I was I would make sure the next time the building is on fire, I would be out before my boss

LOL

In reply to Well, there was this one time…

Well If I was you I would make sure the next time the building is on fire, I would be out before my boss

I Killed my Motherboard

In reply to What’s the worst mistake you ever made as an IT pro?

Back when I was first a tech in the 1980s, I worked on Apple IIs. One thing I knew as a newbie tech was that chips on the motherboard had a notch on their tops to show you what direction to place the chips. All the chips notches should be on the same side, I knew this if little else.
Always there is an exception. Apple IIs where the exception. Their processor could be upgraded. Many people did this. When taking out a chip you did’nt look at the notch. When putting it in you did.
I lined up the notch on folks apple IIs being very careful that all the feet where in all the proper holes.
I would turn on the machine to test the new processor and they would immediately burst into flames, well, they would smoke a little first.
For some reason the notch should be on the other side but I burned three mother boards learning this little nugget.

I did that

In reply to I Killed my Motherboard

I was replaceing a stick of RAM and was garenteed that is was the correct stick for the PC. Needless to say it was not but it did fit in the slot just a bit tight. I knew it was the wrong RAM when I smelled that familar sent of burnt motherboard. That was a fun night.

Deleted the entire CNET.com docroot

In reply to What’s the worst mistake you ever made as an IT pro?

This was back in the day, I’m thinking maybe 1998 or thereabouts. I was working as an ad support engineer here at CNET, and one of the tasks was to periodically go into the directory on the docroot where we stored the ad gifs, delete them all, and then rsync the latest set of gifs from the ad server.

For reasons I don’t fully understand, instead of typing “rm *” I did “rm /*”. Essentailly I deleted the entire disk drive containing our docroot.

Luckily of course it turns out in Unix (and I think most operating systems) a delete like this doesn’t really delete anything except the link to the directory. Our senior unix admin, Ken Emery, was able to restore the docroot with a couple keystrokes.

But man was I in a panic there for a few minutes. It’s one of the reasons CNET instituted a policy where only Ops engineers have the logins to live servers. Us application engineers have to file deployment requests thru Ops to get stuff pushed out to live.

Did something simlar

In reply to Deleted the entire CNET.com docroot

1993 or there abouts. On a Sun workstation. Was just learning UNIX and decided to do a rm -r on a directory I didn’t need while I was logged in as root. Turns out I wasn’t in the directory I thought I was in. I was in the root directory. Goodbye Sun install. Turned out that this particular workstation, I think it was an IPC, required a bootfile to be in a certain spot on the disk. So the backup, located on another drive, did not even help. It took me 2 months to run down the hardware to reinstall. I was at a remote site and had no CDROM. Learned a lot off of that one. Put rm -i in the root profile, check your directory before using that command, don’t use that command if you don’t know what you are doing. : )

Starting the Production Cycle

In reply to What’s the worst mistake you ever made as an IT pro?

When I was a new programmer I ran a test job and used the production name instead of a test name. I did not know that the scheduler would see this as a true production job and start all of the nights processes. Fortunately, the scheduler was shutdown and recycled before any damage occurred.

dunno if it was the worst but sure a goody

In reply to What’s the worst mistake you ever made as an IT pro?

we needed to do a data preservation on a hard drive for legal purposes. The PC was placed on a rack but not marked as do not touch. One of my staff grabbed it and assigned it for redeployment. Went to the techs and of course they had ghosted the SOB. Data recovery did little ( as you can imagine). I got written up for that one.

Hard Drive Replacement

In reply to What’s the worst mistake you ever made as an IT pro?

Back when I was new at using GHOST I had a client who wanted to upgrade a 20 GB HDD to a 40 GB HDD. Long story short I ended up GHOSTing the blank 40 GB drive over the full 20 GB drive. Thus I began living by the motto: “Before you move forward make sure you can go backwards.” The client I had was pretty cool about the whole thing. He had actually backup but his Quickbook databases before I hammered his drive. The reset of the data was stuff he said he could live without.

Network totally unavailable dude

In reply to What’s the worst mistake you ever made as an IT pro?

How ’bout the time when I slapped a desktop drive into our (then) Novell production server and THEN went to run the backup before I got a replacement drive. Bad idea…backup, what backup? Data, what data? Then corporate sends out Mr. ECNE to tell me how dumb I was…ouch, okay…never again.

FORMAT C:

In reply to What’s the worst mistake you ever made as an IT pro?

i was about to have my lunch then, when a user called me to fix his emails. seeing that it was due to software problems, and wanted to fix it in an easy way. i immediately planned to format the C: drive, have my lunch, then when i get back, run Norton Ghost (since i normally create a backup system image per PC). Only to find out that what i formatted was the D: partition, which contains all his documents, including emails! i forgot that after booting from a W98 bootdisk, the C: (NTFS formatted) can’t be detected, and the C: now is the D: partition. … The sad thing is…. the user had no backup! 🙁

Not being better prepared for OS change

In reply to What’s the worst mistake you ever made as an IT pro?

I’m the general IT person for a very small company and a couple years ago we decide to upgrade to a couple new servers and Server 2003. We chose SBS2003 for the DC (we were running NT4) and Win2003 server for the file server. Most of these final decisions were made by the boss’s bro who has a PHD in comp sci and has had an attitude about me since he came onboard (I’m self taught in computers). He is notoriously cheap and thought that saving some bucks by getting the new SBS2003 in the Dell bundle would be a good idea.

I’m careful about backing up and CYA whenever I upgrade anything so I read a lot about the new 2003 OS’s, and went to a couple of free workshops. Because we’re small we don’t have a lab for testing things like this, although I often will on my home lan which runs Win 2000 Server. But I just didn’t want to put an eval copy of 2003 on my home lan because then I’d have to replace it later with all its attendant crap. I thought I’d cut it OK since I had no problem installing NT at work and Win 2000 server at home.

We buy the Dell hardware with the preinstalled OS’s. The file server setup went fine; we weren’t using AD with that one (Those files were on our original NT box which was our PDC as well.)

Now the DC upgrade had to be scheduled for when nobody was using it (which was just about never!). I was going to do it while folks were away at shows since that was the least busy time, even weekends were bad, and show times had 4 days with little network use. So a show period rolled around and I began. I had decided to do the entire change from scratch including all accounts since the NT was so old. Also I wanted fall back in case things went wrong; I could go back to the old server with its account information serving the network.

So… a long line of problems started, separated by long periods of waiting for a ‘not very busy’ window to finish the job…

First, Disk 3 of SBS2003 was buggy (taught me again not to be first on the bandwagon with MS, hadn’t done that for years) and bombed the preinstalled setup, and I had to wait 6 weeks for a new disk from MS. Back to NT…

Then the second try bombed because of some network problem setting up the clients which I still can’t figure out what went wrong, but made me reinstall…with the new Disk 3…and it still wasn’t working right, and I ran out of time…

Three times was the charm; the third time which was 6 months and three shows later all went well and things have been OK since then.

I couldn’t believe that ancient NT PDC came back every time I replugged it into the network… I was prepared for major restores every time.

Of course the the PHD was my supervisor and was not pleased by all the delays, which was duly noted at my next performance review even though I had not lost a scrap of data during the long and painful process…

Next time I will insist on some lab testing first (I’ve been saving our older unused machines for it since then).

Not one by me, but…

In reply to What’s the worst mistake you ever made as an IT pro?

we had spent months figuring out what we needed for our first LAN. On Friday afternoon we had it all ready for order on Monday morning. On Monday, the deal was off. The boss’s idiot son had been talking about it a church to someone who said he could take 10% off the best price ans still deliver the same quality, etc. the $$$ won, and the computers eventually appeared– without the DOS installed on them (part of the agreement). It took weeks more to get all that weeded out and get the rest of the equipment in we required. And the guarantee on the parts? Well, that bad motherboard still has not been replaced– after all, the vendor said, you get what yu pay for…..

Backing up Home PC

In reply to What’s the worst mistake you ever made as an IT pro?

I nearly lost all my data (documents, contacts) from the home PC, as there was no backup done before. Luckly I could get back most of the information.

I didn’t think it would reboot NOW!

In reply to What’s the worst mistake you ever made as an IT pro?

In my first week of desktop support for a large company, I was asked to put a copy of MS Project on the corporate file server so that users could run the install to their PCs. I mistakenly installed Project on the server and when it was done, it rebooted the server. The server that almost every one of 500 employees was connected to at the time! Yeeeesh! that got a lot of people into the computer room in a hurry…

In alphabetical order or by severity of screw-up

In reply to What’s the worst mistake you ever made as an IT pro?

If you work in the industry long enough, you will absolutely have some world class screw ups.

1. Back in 1998 or 1999 (I’ve blocked the date from my memory), I was working on an 8 station Win 95 peer to peer network running Peachtree Accounting software, with the database being on the secretaries drive. They bought another Win 98 desktop to serve as a print server and file server, which was just basically a PC that no one used with shared folders and printers. I set it up, copied the database over to the new machine, and pointed all their Peachtree software to point there, and all was well with the world. One LITTLE problem; I forgot to move the backup software (a Syquest drive, remember those?) to point to the new database to back that up. SO, what ended up happening is that it backed up the old database that no one used over and over again. 3 months later, you guessed it, their new database developed a corruption in it that was unfixable. I showed up, went to restore the backup to the server, and wondered why the date on the file was 3 months old. They spent the next few weeks pulling up the paper invoices, and re-entering all their data. Woopsy!!

2. Later that year, at the SAME COMPANY (they loved me after this one), there was a lightning strike on the building. It fried most of their network cards, the hub and a few video cards. While I was replacing the hub, they decided now would be a good time to upgrade it from a 10BaseT to 100BaseT. No problemo, the nics were all 100BaseT compatible, so just plop in the new hub and voila! Ready to go!! Ummm… no. When I had first put together the network, I had used some of the old network wire from their Novell 2.x network. I had put Cat 5 wiring to add new machines that weren’t there before to the network. However, the old wiring was Cat 2, which can only handle 10baseT connection. When I tried hooking up the 100baseT hub, the machines were getting/losing connectivity faster than I could keep track. It took me a full day to figure out that I shouldn’t have used the old wiring. Needless to say, they never called me after that.

My knowledge and troubleshooting skills have greatly improved since those many years ago.

Missed one little check box

In reply to What’s the worst mistake you ever made as an IT pro?

I work at a Christian Camp and Conference Center that has a network of 4 servers and over 50 clients. We are a pretty conservative bunch of people and were excited about the new features of our new antivirus software.

When installing and configuring Trend Micro’s Client Server Messaging antivirus suite, I waited to activate the new e-mail filtering until after I made sure the desktop antivirus and Exchange antivirus systems were working.

One day the executive director got a pretty graphic pornographic spam e-mail, and he told me that I needed to get the filters configured and working “TODAY”. I quickly glanced at the manual and configured it that day. Everything was fine, but a not perfect. I went in to tweek it a little and saw a check box (with a check in it) that said “match must be exact”. I thought that if I unchecked that box it might get a little more selective. I went ahead and unchecked it and went home. That night I e-mailed a file to my work address and got a strange notice back that the e-mail I had sent contained language that violated the organizations e-mail policy and that it had been deleted.

Early the next morning, I got a call that our e-mail was down and that we were getting all sorts of calls from Pastors, Bishops, Board members and a lot of Donors that had the same message I had gotten the night before. They couldn’t figure out what “offensive” language they might have used in an e-mail. Some hadn’t contacted us by e-mail in months!

Come to find out, not only did the filter stop incoming e-mail, it also went through the whole exchange store and deleted any message that it thought was offensive. It sent a copy of the message to the sender, recipient and administrator accounts. By unchecking the box, the filter was now seeing words like “Ass”istant, “Ass”ociate and hundreds of other words as offensive. That was bad enough, but the notification had a word in it that the filter thought was offensive so it deleted the notice, sent a message to the sender, recipient, and administrator – a never ending loop that literally sent out hundreds of thousands of e-mails before it crashed the server. It also deleted attachments and e-mail that people had saved in their exchange account.

It took me almost a week to restore “most” but not all of the old e-mails and attachments and to delete thousands of notices from the administrator account – all because of 1 little check box.

One good thing did come out of it, no one ever rushes me to implement, install, or modify a new program until I have a chance to learn about it and do a test install!

Not your fault

In reply to Missed one little check box

Dear colleague,

Don’t you think you have fallen into an idiotic program design trap? It is not your fault at all.
A lot of antispam software is the same way faulty. Most of the so called developers have not any linguistic literacy but write linguistic filters. They can code but not design… The box you mention is an example of stupid design.

Formatted a DAT tape to copy a backup

In reply to What’s the worst mistake you ever made as an IT pro?

Except I formatted the backup itself. DOH!

It was one of those, look around and if nobody’s lookig just restore a differet backup.

“Your file is missing? Are you SURE it was saved properly? Oh, thats too bad, I’ll see if I can find it, but no promises…..oh that’s okay, you’re so very welcome, it’s my job afterall. No don’t worry about it, okay I’ll let you buy me ONE drink but I am not sure if I can restore it or not. Just for my troubles? Oh, how kind of you! Do you have any girlfriends that want to join us?”

Then there’s the time I sent out copies of a George Carlin standup special (VERY filthy!) to clients waiting for promo material, instead of our very own Julian!

re:

In reply to What’s the worst mistake you ever made as an IT pro?

Hiring a newbie who said he was the best to configure my linux server which crashed taking most of our clients data with it… major $$$ payouts

Remember the Network/modem cards?

In reply to What’s the worst mistake you ever made as an IT pro?

I forgot all about them just the other day . Customer had guts of a Compaq torn out and reinstalled to a new Intel box some time ago. He managed to crash and thrash the W2K that had been installed.
I get it , no problem reinstalling W2K, huge problem trying to figure out what driver to get.Spent more than a few minutes trying every single freakin Compaq network card driver.
Fellow tech arrives on scene, pops in a modem driver. voila, new device found, nore more pci device? in device manager.
Man I felt so stupid , I walked outside for a few minutes , lit a marlboro and thought about becoming an MTA employee.
Customer was waiting too, good thing he wasn’t all pissy about it….but then again I’m not the one trying to use AOL dialup either.
Bottom line is, sometimes those freaky little cards got secrets….I never wrote down half my stuff on em. Any bets on how many notebooks I bought that night at the dollar store?

New DBA

In reply to What’s the worst mistake you ever made as an IT pro?

When I was still fairly new to being a DBA I once changed the quantity on hand of all parts in our parts master table to “3”. That was almost a million records. 3 cabs, 3 nuts, 3 washers, 3 engines, … you get the idea…..On DB2 for MVS it took mere seconds….and hours to recover!

George

Historic Blunder

In reply to What’s the worst mistake you ever made as an IT pro?

1969. Largest computer in the U.K. (360/195, UK Atomic Energy Authority, Harwell). Submitted reactor design program (FORTRAN of course) with expected run time 24 hours (no multiprogramming). Forgot to include checkpoint card in deck. Run failed after 23 hours so had to restart from beginning. Moved to Australia.

Taking down the porn king

In reply to What’s the worst mistake you ever made as an IT pro?

I reported a manager when I discovered a huge cache of porn on his hard drive.

I was tracking down the source of a virus that our firewall should have stopped. I found out this guy had downloaded one of those self-install dial-ups that connect through a premium rate number, which was costing the company a fortune.

On another occasion I might have just have had a quite word with the guy. However, when I was called out to support his computer, he ranted about how lousy the tech support was and how none of us knew our job.

I took the information to the HR department. 30 minutes later I was in a meeting with my line manager and the manager that I had reported, being told to retract or my actions would be the worst thing I had ever done in my career.

I refused.

The manager was dismissed and the CEO sent an email throughout the company reminding everyone that this behaviour was strictly against policy. I received a letter commending my actions for “maintaining the highest company standards”. Rumours circulated that after the manager had gone several scams that he had been involved in came to light.

Several female colleages approached me to say what a relief it was that the guy had gone. It apears that they had attempted to report his “inappropriate behaviour” on several occasions.

One month later I was formally disciplined over time-keeping. I had been late once, a rare occasion, but my line manager claimed that he had verbally warned me about it several times.

Three weeks later I was given a final warning for poor performance, and six weeks after that I was fired.

An industrial tribunal found that I was unfairly dismissed, and the compensation paid for some intensive courses. Now I am a programmer, earning 30% more, and my tech support background opens doors that wouldn’t be available otherwise.

I confess that I reported the porn out of resentment for his attitude, not out of a sense of moral duty. I am not proud of what I did, because I did it for the wrong reasons. I lost a job that I was happy with, with a good company where I had lots of friends, and money can’t replace that.

But what happened was a wake-up call in many ways.

First, tech support is a job, and now I have a career. My daughter (almost four months old) will have better schooling and better chances than I had.

Second, people like that manager are not nice. They will “like” you as long as you support their actions and attitudes, and dump on you when it suits them. Every time I turned a blind eye, had a quite word instead of reporting them, I became complicit in their actions. Never again. Seeing how releived some of my colleagues were when this guy was sacked has opened my eyes to just how serious these things can get.

So, I have lost my happy ignorance, and my complacent but contented “just doing the day-to-day job”.

I have gained is a new awareness of the world, and my place within it. I have a list of courses, both academic and to develop myself as a person, that appears to be ever-growing. I am no longer happy with where I am in the corporate tree, seeking the position that really fulfills my potential, provides true satisfaction. And I now know that I have to make up for my previous uncaring moral attitudes, if only for my own peace of mind. I am a better man now, but it is hard work.

Sometimes, when I look at what I have ahead, I wish I could turn the clock back, unlearn those lessons.

Power means LOUDER??

In reply to What’s the worst mistake you ever made as an IT pro?

Well, this wasn’t when I as a full fledged Telecom Tech but it was a big “Oh shirt”
When I was going to school for Electronics, learning or gates, nor gates, and gates and nand gates, Bolangela Alegbra and all sort of thing I’d never use in my livilihood. I got a summer job working at at computerized radio station.
What is a computedized radio station? Well. its a radio station that works off tape drives and a computer program that activated the tape drives to start when a beep code was feed to the computer from the preceeding tape drive. Pretty simple once you understand it. One of the tapes was a dis jockey’s voice that introducted songs, gave generalized info of the weather and time of day. All that was recorded at the station was the commericals. My jobs was to change tapes and and time id’s when they reached a certain point in the program. I could also program the computer to signal different tape sources to activate out of sequences.
Here’s the blunder, this was a “Top 40” format and I had a tape that was more of “Howling Wolf” format, I think the song was “Red Rooster”. I setup the tape to play over the air but when I re-recorded so it would fit in a tape drive that played over the air, I unknowingly used the power gauge as the volunme gauge.
I want it to be load so I cranked it up to +10-15 DB!! Well as soon as that tape was signled that “Howling Wolf” ran up the wire to the transmitter on the mountain top and fried the front end of that reciever to a cripy golden brown. We were off the air in a nanosecond.
I blame mother nature lighting strike of course and got away with it. If your out ther mother forgive me!!
But it was weighing heavy on my conscience ever since. I won’t say what the stations name was or what part of the country this happen in but want to thank you for allowing me to vent my guilt on “Tech Repuublic”..

Regards,
Power Howling Wolf

Worst Mistake…taking a job i didnt want

In reply to What’s the worst mistake you ever made as an IT pro?

I am currently in a job i didnt originally want. One of several worst mistakes. None of the other people like me, my manager will not give me a reference when i leave. I don’t believe i did a bad job even thought i was over qualified. But still, it leaves me feeling that i can’t take another position like this ever again. And i would never come back to this company knowing what these people are like and how the company business works.

A small consolation………..

In reply to Worst Mistake…taking a job i didnt want

One thing to consider: At least you HAVE a job! I became unemployed two years ago when my former employer became insolvent, and even with 30 years in IT, I have only had three interviews! Of course, only having an AAB degree seems to have caught up. If your resume doesn’t have BS in CS/MIS and 20+ certs, employers will overlook you.

worst thing I ever did….

In reply to What’s the worst mistake you ever made as an IT pro?

Accept my current job. It is the most boring and annoying job I have ever taken. All i do every day is drool on my keyboard until someone tells me the medical software crashes or they have lost connectivity to the server. There is nothing wrong with my setup and I have had several other IT pros come and look at how I setup the domain ad they agree. Medical software is programmed with intentional bugs so that you keep your contract of software support from a vendor that specializes in that exact software. Now Im getting pretty savvy with it after about a year but when things go wrong it give me error prompts with codes telling me to speak with a logician person or GE. Im not in over my head on this one, Im just swimming in crap!!

Losing my temper to incompetent vendars & users?

In reply to What’s the worst mistake you ever made as an IT pro?

I always loss control to my temper and start jumping on the Vendors & Users, when ever they repeating their mistakes.

I realize that my actions actually never help me to solve the problem?

Again I have to sweet talk with them and ask them to be more alert?

And again, the mistakes keep on repeating?

And again, I loss control…

Where to begin….

In reply to What’s the worst mistake you ever made as an IT pro?

I guess my biggest blunder, happened when restoring a file from a tape on a SCO Unix system. The tape prompted me for a file name to restore the file as, when it found the file to restore, and instead of hitting ‘.’ which signified the same name, I hit ‘..’. Well I was in the root directory with admin access, and .. is a special file which referes to the parent directory. Renaming the file I was restoring to .. made it so that you could no longer use the directory structure without fully pathing every file. My boss was away, and that was the worst phone call I ever had to make. He was calm, and in about 1/2 hour we figured out how to copy ‘.’ to ‘..’ and evrything was back to normal.

Calling out a admin

In reply to What’s the worst mistake you ever made as an IT pro?

Telling the rude administrator at a previous place I worked that he was being rude…. He got me fired >.<

HALON, need I say more….

In reply to What’s the worst mistake you ever made as an IT pro?

About a dozen years ago I was working in a large mainframe shop that had a two large 80 gallon HALON tank system to protect the computer room in the event of fire. Every night we had a break of about 2 hours where there was literally nothing to do, so to pass the time, my partner and I got into an escalating rubber band fight and we were looking for larger and more powerful rubber bands to fire at one another. We ended up settling on these foot long bands that could easily sail across a 45 foot room and hit you with enough impact to leave a nice red mark. When hit up close, these things would leave welts and bruises.

One night, I was leaving the computer room to go get a soda, and my partner attempted to fire one of the rubber bands at me from a distance of like 10 feet while I was walking away from him. He missed me and accidently hit the unprotected HALON release button.

After the alarm sounded, and everything shut down, and the 160 gallons of Halon were dumped into our glass enclosed area, it was extremely quiet after we returned to the room. Every report and unbound document was blown everywhere, and we printed about 4000 pages every night. It cost the company upwards of $10k to get everything back up and running, and get the tanks refilled.

The funny thing was that after we got everything cleaned up and back online, the first thing that they did after we refilled the HALON tanks was to requisition a $40 plastic cover to be installed over the release button.

On the day that they were mounting the cover, they maintenance guy drilled right into the two wires and shorted the system, triggering another HALON dump. Suffice to say, they took our rubber bands away.

Halon and 2 techies I knew….

In reply to HALON, need I say more….

I could see the 2 techies I knew pulling a stunt like that! Thanks for the Laugh! 🙂

hardware and software blunders

In reply to What’s the worst mistake you ever made as an IT pro?

hardware- I wanted to make an expansion card with fans on it to plug into an ISA slot on an old computer with a VLB bus. Using a voltmeter I found out the ISA slot had a positive voltage pin adjacent to a ground pin. Killed everything on the system except for the SIMMs and the hard drive.

software- made the Power Users group part of the Administrators group. Try logging on locally like that. All you have to do to fix the problem is to reinstall the server. I tried all the remedies I found on the internet without success.

Forgot to re-start a service on Server 2000

In reply to What’s the worst mistake you ever made as an IT pro?

I got fired because of what happened to the mortgage office due to the fact that I forgot to re-start a service I had killed in order to get something done. Mortgage people have sticks in their buns about this stuff.

then i was a newbie

In reply to What’s the worst mistake you ever made as an IT pro?

emmm i was a newbie in this auto mfg comp.the internet suddenly downed.all config in the fwall were erased..becouse of misspressed button..it all seems the same..i spent 4 full days reconfig it realising that it was never backup by my partner..and my housemate said “i thought u never be back and im going to send u ur bed”

never underestimate the value of testing.

In reply to What’s the worst mistake you ever made as an IT pro?

Not my mistake, but I was the one to discover it.

In the mainframe COBOL world, there is a ‘STOP RUN’ statement. Similar to an ‘END’, it terminates processing nomally. You can also code a ‘STOP XXXXXXX’ and ‘XXXXXXX’ displays on the system console – this is usually used to indicate some sort of error.

Some joker in our shop coded ‘STOP THE ROMULANS HAVE ZAPPED WORKING-STORAGE’ in the middle of a program. He put code to branch around it – so in theory, it would not be execute.

Some other joker, making the proverbial one-line change, made a change, compiled the program and cataloged it to the production library – all without testing. This program was in a critical nightly nightly job.

That night, the operator (me) was surprised by the strange message that printed on the console log that night:

STOP THE ROMULANS HAVE ZAPPED WORKING STORAGE.

The oncall progammer had to come in and figure out the screw-up. This was not the person who made the change.

The moral of the story is to TEST EVERY CHANGE – no matter how small.

Dropsy

In reply to What’s the worst mistake you ever made as an IT pro?

Not my mistake but, an intern was moving a production server from the test area of the server room to the production area. When lifting the tower style server to the top rack he stepped on the power cord still attached to the power supply and the server went back over his head and crashed to the floor 6+ feet. It was hard to believe that the server started right up after every card, processor and memory chip were reseated. This was about 5 years ago. Lesson learned: Always unplug every cord from the server before moving it.

Before most of you were born……

In reply to What’s the worst mistake you ever made as an IT pro?

I was working with good old big iron at a bank in NY in the 70’s, and I had to verify the patch level of an O/S using PDZAP (any fans?) I had to use the console to verify this information. After I was done, the printer gave me a report on what I had looked at and I went back to my office.
Several months later the drive that the page-data-set was on, went casters up. The CE’s were called in to repair the HDA, then after it was repaired, the operators presses the IPL button. After a few seconds, the system just stopped as it was initializing the PDS (Something real computers just doesn’t do, stop) Of course I had to go in at 3am to see what the issue was, and I was not letting the CE leave without it working. Working with the problem, the CE decided to call in his boss to fix it, HE found NO TROUBLE with the drive, yet I can’t IPL?? I attempted to restore a copy of an O/S many months old, and it finally worked! so I verified it was not a hardware problem, but I was NOT restoring and O/S several months old, so a call was placed to some PSR’s (remember those?)that I had known. It was advised I open up an incident so they can come take a look see.. Well needless to say, I had a room rull of PSR’s because after all, this was the nations LARGEST bank being affected. I swore up and down that I did NOT make ANY changes to the O/S that would cause it to fail. After 23 hours of the machine being down, I walked back to my office while the PSR’s were trying to figure it out, I went through some listings and came across the output from the PDZAP………can you guess what I found? I inadvertantly patched an area on the disk, where in the code, it was initializing the PDS! At this point in initialization, the O/S had no choice but to stop, because the O/S was not fully loaded and could not hanlde an error like this (invalid op code). I walked back into the computer room, across the raised floor and over to the head PSR, “I found the problem…” (we have all been there) they all looked at me an laughed out loud and said “At least we had the chance to all get together, it was fun, and NEVER call us again!” Now each time I had called for support, since they new me by then, I got asked the same question each time I called: “Have you changed anything?” hahaha

Kids & CPU

In reply to What’s the worst mistake you ever made as an IT pro?

I had a clients pc at home in my office with the cover off and cpu on the desk. Walked out for 5 seconds, reurned and no cpu. Figuring my bad memory had come into play I looked everywhere for it though I was sure it was on my desk. Went to look around the house, who knows, maybe I took it with me. I walk into the lounge room and found my 2yr old happily smashing the pins flat on the coffee table with his plastic hammer daddy bought him just that morning. My IT mistake? Thinking that computers were safe with the office door open for just 5 seconds.

Worst & Dumbest ever!

In reply to What’s the worst mistake you ever made as an IT pro?

1. Not saving a WEEK’s worth of continuous data typing for my Thesis + Power cut. You can imagine the loss.

2. Telling my boss to F*** off.

Using Scrapbook FF addon to copy site…

In reply to What’s the worst mistake you ever made as an IT pro?

Yesterday, I did this. This is the worst because I lost every single file in my account – including other people files in my account. And managed to wreck havoc against cPanel.

I used Scrapbook, a Firefox extension that got great reviews. The problem is that the extension does NOT get along with cPanel. I used it within cPanel while logged into my account. Scrapbook, instead of only coping all the pages, it dumped everything into trash. The problem is, I can’t restore data by clicking on them in trash. Scrapbbok also caused a .htaccess.html file to store itself in one of my folders. I had to put in a support ticket/forum help to get the President of t35 to bale me out.

I’m sick to my stomach about this because it involved other people’s files as well.

signed tpgames, a webidiot as webmaster does not currently apply.

Becoming a manager

In reply to What’s the worst mistake you ever made as an IT pro?

I know this thread is old, but I could not resist sharing. The headaches of being a manager was just not worth it to me. Now I’m trying to get my career back on track by learning new skills and hopefully getting back to what I love..being a hands-on techie!

One time at band camp

In reply to What’s the worst mistake you ever made as an IT pro?

Actually Banyan….

We were putting a new IBM server on the network to replace an old Vines server. We had done up a complete (I know…it wasn’t) checklist for the process, so when the contractor brought it in and gave us the checklist, we didn’t check it over again. We connected it to the backbone and Banyan’s Streettalk database went nuts. Seems that Wang had given the new server the same name as the one it was replacing and the network went postal. It took 36 straight hours, 5 full installs of the Vines OS and backup restores before it was back working properly.

I now triple check everything and take nothing for granted.

[Author]

Replies