Discussion Locked

What's the worst mistake you ever made as an IT pro?

0 helpful
240 replies
Closed to new repliesThis discussion is closed. It stays readable, but it takes no new replies.

We talk a lot (and laugh a lot) about the dumb things our users do--but of course, we do plenty of dumb things ourselves.

The recent article (https://www.techrepublic.com.com/5100-10877-6044807.html) and download (https://www.techrepublic.com.com/5138-10877-6040785.html) "10 classic clueless-user stories" showcases bizarre tales of user behavior as described by some TechRepublic members. Now let's turn the tables and talk about OUR mistakes.

What's the biggest blunder you ever made (or almost made) during your IT career? Don't hold back! Share it here or pop over to my blog and see what people are saying.

Jody

240 replies

by dc guy

The worst mistake I've made in my career is one that I've made several times: trusting people who turn out not to be trustworthy. I've lost a couple of jobs by being the fall guy. I've made some decisions based upon what I was told by people I trusted which ruined me. I'm still paying for them. I should be retired by now but I'm still working because of misplaced trust.

Still, I can't get over the fact that not ever trusting anyone has got to be an even more dismal life.

by neil higgins

but I suppose my big time "gaff" was asking someone at work to delete files X,Y,and Z off one of the desktops,whilst I went to lunch.Dont worry,I said,they're not important,as I've finished those "work-tickets",so there's no point keeping them on file.One hour later,I returned to find the "said person" had misheard me,and was formating the entire hard drive.Forunately,everything had been backed-up elsewhere,as disembowlment would surelly have followed,not to mention my P45.My boss however,was still not amused.Guess what I spent doing all afternoon?

by ahunter6

in my first reall job, hired straight out of internship, i decided that i neither had to back up the boss's laptop, nor actually follow the instructions for putting in the new pcmcia nic. i simply popped it in, ran the install, and it all went to hell.
This was on Win95, by the way.

ended up having to wipe the device from scratch because so many thing failed after that. this was before i knew about imaging, before i knew just about anything really. i had only ever worked on my own, single machine at home. I was THIS close to getting fired.

by dhawthrone

When I started at my current job, our Active Directory was mainly used to for access to an intranet site and none of the client desktops were Windows 2000 or on the domain yet. I downloaded a software package of "NT Tools" and accidentally set everyone's password to expire. Of course with no way for users to reset their own password, i spent the next three days managing "special" voice mail box setup specifical for people who need to login, calling them back individually to give them a new password. Great way to filter out unused accounts.............

by stan20

My biggest mistake was trusting non-technical people, who were brought in to handle sales and business managment. They are retired and rich now, and I'm still working for a living.

by gsg

I insisted that a PC I was working on had a bad power supply. I filled out the work order (warranty), tried fixing it myself, called for advice from a co-worker, looked on the back for the serial number, and called for repair. When I was moving it to put it on the repair shelf, I realized that I'd made a terrible mistake. It wasn't plugged in. Imagine, they need electricity to run.

Wouldn't be so bad if I could have kept it quiet. No, I had to call and cancel the work order, and they wouldn't take "I fixed the problem" as an excuse. I should've just opened the case, and ripped out some wires and turned it in for repair.

by i1-14174

Did you ever had the need to delete some old movements records from a couple of table in a live (worldwide running TP) DB ?
Double check and have someone looking from your back to avoid to mistype "where year=2005" with "where year=2006" !!!

After a couple of hours, a terrible belly ache and thousand of angry phone call, I recovered almost everything from the night backup. Whew!!

by thefraz

IT rule #1 was obviously ignored--always check the power plug before you call for repair.

by mjwx

Which wasn't that long ago for me. I made the mistake of switching the little red switch on the back of most PSU's from 240v to 110v. Not only did I fry the power supply but it also tripped the RCD taking out power to the entire wing.

Fortunatly no PC's were harmed in the making of that mistake.

Mike

by takalm

I deleted the CEO's mail box after recieving the wrong description of the account that was supposed to be removed from the network.Luckily managed to recover all the stuff back ,although i had a good explaining to do to the big man himself as to why he could not log back in the network.

by jeff.lane

My first job out of university, taken on as a developer for a large logistics company and the IT Manager for some reason told me to start ripping out all the old cabling! Suffice to say his instructions, 'to pull and cut all the thick cables' led to numerous calls to the helpdesk from irrate users. To top it all I also managed, whilst cutting cables in the comms closet, to lean back onto the main power supply and shut down power to the whole building!

I left after two days and put it down to experience!

by atnixon

When i first started in IT, i came across a server that was running out of disk space, exchange would not run...So, i deleted loads of LOG files....including all the exchange server log files...DOH!!!!! Hence, i have never done it again....

by james

I once deleted our entire accounts data after recovering everything from a damaged server after a flood.

We got all our data back and I copied everything over to the other server fine. The Sage install lived in a folder of old files from the previous server. After rebuilding our file system and copying over everything to the new machine, I "forgot" that I'd patched in our accountants to the old data in this temp folder, even though it was newly installed. Having spent a few weeks working through our backup system and checking everything was fine, I cleared out the "spurious" data. Then swore quite a lot when I got the first phonecall.

by philip_jones2003

You are not alone in blowing up power supplies. I did the same on my own pc. Cant for the life of me remember what prompted me to do it except that it was there and I swear was winking at me.

On reflection though, its a pretty poor design that cant detect the correct voltage and ajust accordingly.

Knew of an electrician that installed an emergency power supply for four AS400s. He decided to demonstrate just how 'uninteruptable' it was by cutting the mains supply in the presence of the IT director.

Nothing happened of course, and a number of things stopped happening, including the four AS400s.

Story goes that the director chased the electrician around the site with a very real threat of violence.

by cedrics

A guy in my dept. at work did the same thing during the replacement of one of our small file servers.

I can tell him this..maybe he'll feel better.

by steve

I think the biggest gaff I made was when one of our customers purched a new external RAID box to go on their Exchange Server. I was asked if I could go in and configure it for them, I said no problem, I will pop in at the end of the day and set it up. After plugging in the external box and powering up the Compaq server (they are nice and easy for configuring RAID) a message appeared on the start up screen that the system had found new drives and would I like it to initialize these disks. I pressed "y" then immediately thought "no I don't". Too late, I had initialized the whole RAID including the system. The first indication I had that the customer had come back in the room was, "Does oh bollox mean that didn't go quite right?". I then spent until gone midnight re-installing windows NT4 and Exchange Server and restoring all mail etc. from the previous backup, which fortunately I had done just before plugging in the new box.

by steve

Next error, the MD of one of my customers had a dodgy hard disk in his desktop pc, so I got a new one of exactly the same size and thought I would ghost his original system to save the time of re-loading th OS and re-installing all his software. Then, promply overwrote the original drive with a blank one! Fortunately no data saved on original drive, so did end up spending the time re-installing everything.

by double debo

Works great for mistakes just as you discribe. Which I have made on more than one occassion.

by cpromptservice

I did something simular the other day. I had 3 identical machines to build for a customer and figured I'd use ghost to speed things up. I built two simultaniously and went to ghost the 3rd HDD, and oops, ghosted blank drive over imaged one. Turned out HDD was on secondary IDE Bus. Oh well I thought, still had one machine with a good image, but I forgot I moved the imaged HDD back to Primary Bus, and as I pressed enter on Ghost I realized I just asked to Ghost secondary drive to primary.
3 Hours to image the machines, 3 minutes to wipe them and another 3 hours to reimage again. I have since learned that it helps to use a spare different sized HDD to differenciate.

by pkr9

Teaches you the value of backups....
It has to be tried befor you really understand how usefull that hour while backup is running is.

by steve

I totally agree about backups, I am always on at my customers to do this. But who was the one that lost all his data after a hard disk failure with no backups, yup, me!

by jacob.steenhagen

We used to have a policy that email did not get back up, period. The exchange server was backed up, but not the mail store. One day at lunch I decided it was time to update NT4 to SP6. It was a relatively simple thing, so I figured I'd be safe. The service pack applied just fine and said it had to reboot (as is always the case with Windows). This is where my nightmere began. The server refused to come back up. It was start loading NT then blue screen long before it was actually up.

Not having a backup of the email, I really didn't want to just do a tape restore as that would result in loosing all the email history. It was after a few failed attempts with command line utilities that I remembered that we had mirrored disks. So I broke the mirror and physically removed one of the disks. I then did my tape restore, put my mirrored disk in the server as a "d" drive, and used a command line utility (ERD Commander, as I recall, but that could be wrong) to copy the exchange databases over. I then took the "good" (yet really bad?!?) disk back out and booted up. Fortunately, that worked as the next step was to loose everybody's email.

Shortly after that incident, coperate changed their policy and we now backup mail once a week.

by avid

did the same thing with a dell server at a bank. wiped all the check imaging. performed a restore to fix it. took two days. the bank manager just laughed at me. he was very nice about my screw-up. he still needles me to this day. he said "it is good to know that you are human." we could use a few more people like him in th world.

by ksand

The RAID initialization burned me once as well, they should really call it a format.

I was adding a disk to a RAID card and ended up initalizing the new disk along with the disks the server was running. This was the centerpiece of the business. Backups saved me; however, there was not any docs on how to do a full restore of a domain controller, exchange, and veritas (all of which resided on this server). After seeing the sun rise and no sleep for 20 hours straight the system was back online. I am glad it happened to me, I was getting a bit cocky with my abilities and this disaster I caused put me in check.

I always have a second pair of eyes available when working on such things now. That one sure beat the hell out of me.

by rgovaerts

Hi,

I managed to crash the Main Domaincontroller at my previous work. It wouldn't have been a problem
if there were a secundary DC, but there wasn't. I was installing a printer on the machine and during the install i got a nice Blue Screen. I can tell you, you completely freak out when this happens. The other employees didn't mind so much. They were all forced to go home early :P. Lucky for me, this happened at 3pm. I was able to fix the problem and was home at 9pm. I was also lucky to find the reason for the BS, so i could defend myself against me angry boss. It was just a matter of time...

by toivo talikka

I was a green salesman back in 1982. The IBM PC had just been launched. My product was the MDBS III database application. After a telephone marketing campaign, the first client turned up. Because the database application was sold in two flavours, the big question was which one of them should the client buy.

I was put on the spot - after a few umms and errs, my definitive recommendation was "...CP/M, because it has been around for a while and we are all comfortable with it, and we have heard that the newcomer, MS-DOS or whatever it is called, requires 128KB of memory to run."

There you go...

by info

I will admit I have made a few.

Just last week I spent a while updating all the service packs/patches on an xp home machine.

It was an ASUS notebook and it had a shortcut on the desktop that said NTFS conversion or something like that.

Without thinking twice I thought "yep thats a good idea" and double clicked it, at the end of the batch file it said do you want to update the pointers or something like that.

A orange light was dimly flashing in the back of my brain and I knew I had a 50/50 chance, bugger it, I selected no.

Restarted the Notebook "please insert system disk"

2 hours worth of work down the drain.

Another crushing stuff up I have made is ghosting what I thought was the source disk (but was actually the blank destination disk).

I ended up with a perfect copy of a blank hard drive.

by orjan

A couple of years ago a IT technician found the time wrong on a server at the stock exchange clearing center in our country. He set it back one hour(he thought) but instead he set the time back by one day. The result: When all the broker systems went in to collect their data all transactions from yesterday had disappeared, all starting prices for all stocks where from wednesday, instead of thursday, and people started to trade like maniacs since thursday had been a really turbulent day. After a while the trading was stopped (panic button :-)). The stock exchange start the trade again late in the afternoon. A whole day of trading missed by a simple mistake by a IT technician.

by per.hallgren

I am responsible for the Student Ldap at a mayor University in Sweden. One time when I tested a new ldap browser I decided to delete some System Administrators that had left the University, a normal update of active Administrators. I marked the account in question and clicked on delete. Wrong place to do that! The entire folder containing the Systems Administrators for the University was deleted.

I had to learn in a hurry how to create an ldif from our secondery master over the deleted folder before they synchronized their records.

by firestorm69

and I have learned many VALUABLE lessons. Here are some of my worst:

1) Wrote an interface for an e-commerce site to their CC gateway and because of the way I chose to check the cards, if the card was good, but the address or CVV2 failed, it would deny the order but keep the hold on the card. The client sold items worth $2000+ and the customers just knew their card was good, so they kept trying, some as much as 5 times. This resulted in their cards having several holds put on them for several thousand dollars each.

2) I was asked to recover a web server hosting 100 domains from a hack attack, with no backups and no records of passwords, domains, clients, or email accounts. Well, all the executables (grep, ls, ps, etc...) had been corrupt, but they had an identical box, so I decided to copy the executables from that one to the other so that I could at least export all the data I needed from their control panel. Unfortunately, I copied the data from the other box too.. Lost all logins and email accounts, but was at least able to save all the sites, then came the process of calling each client and resetting all their account info, which is now logged.

3) Last, accidentally typed "rm -Rf /" on a production server when I was trying to press the right shift key on my laptop and accidently hit the enter key.. Now all my remove commands are typed into notepad first, then copy and pasted into my terminal window after ensuring accuracy..

We all make mistakes, if you do, don't panic. Ask yourself what went wrong, then ask why did it go wrong. Learn from the mistake, and ask what you can do differently to prevent the mistake from happening again.

by pkr9

Anybody can make mistakes. To really wreck havoc you need access to an IT system.....

As a junior programmer in the late 60ties I had a boss with a temper.Programming language was assembler - everything was numbers, and lots was binary code. It created very small and efficient programms, a big contrast of the cut-and-paste, select checkbox programming of today.

Anyway I couldn't make out why I sometimes got a leading CMC7 '5' digit. CMC7 was a 'font' used for magnetic reading of bankchecks, and shouldn't be in files as this was a 'printing only' value. I asked my boss, and he went off on a rave. No help there.
So what do an eager young man wanting to move forward in the world and higher in the payroll do? Make a test just before writing the field, and remove the CMC7 '5' if it's there. Easy stuff, solved the problem immediately.

Accounts then started to behave mysteriously, investigate. Problem was found to be that the CMC7 '5' digit was used internally for negative, unless you were calling the printing routine, where it was used as a digit. I had been writing all negative numbers as positives when updating fields, and subtraction and addition was the same thing.

Needed 3 months of nightly reruns while trying to catch up with real production.

Morale: If somebody asks for help - give it. There's NO stupid questions - only stupid answers.

rgds
Peter

by double debo

There is not such thing as a stupid question. There are really no stupid answers either, there are wrong and right answers.

But the only stupid question and stupid answer is when neither is given.

I have done all of them.

by alxnsc9

I have dared to very politely advise our HQ to reqire contractors rework the "Users Guide" they have written to "Guide to operators" and have even dared to apply an examplary TOC aiming actual activities in detail. Good Lord! So sorry...

by swier miedema

In the early 80's the company I worked at that time installed one of the early Ethernet networks (remember the thick yellow cable?) in The Hague (The Netherlands) The customer wanted the network to be installed in the weekend, so that normal operation could be continued on monday with the shiny new network! So we went and installed the network, tested it thoroughly, and left to enjoy the rest of the weekend (it was a fairly small network).
On monday the account manager called the client to ask how he liked the new network. He said fine, but every quarter of an hour or so, something odd happens, and then we cannot use it for about two minutes...
So we went back trying to find out what went wrong. Everything seemed to be in order, so in the end, desperately, we hung a scope onto the backbone in order to measure the signal. And indeed approximately every 12 minutes we saw an enormous amount of spikes rolling over the network...And we didn't have clue as to where it was coming from....
As the day proceeded, we trying to figure out frantically where this signal was coming from, and, as it was a summer's day, the temperature was quickly rising, so someone opened a window to cool things down a bit.
The sounds of the street were coming through the open window. Suddenly we heard: "Dingelingeling", and over the network another series of spikes was sent, disabling communications over the network again.
Most Dutchmen know that when they hear "Dingelingeling", the electric tram is close by and is warning you to stay out of its' way.
It took us three "Dingelingeling"'s to figure out the connection between the EMC field the Electric tram created, and the spikes on our first Ethernet network...

by lesmond

I took on a project for a firm of lawyers - nothing too big, an exchange server, six client pcs, a wired LAN, and a couple of printers. The staff in the office were basically computer illiterates, calls ranged from "I clicked in "record a macro" and don't know what I'm doing" to the final straw when I got a telling off over the phone at seven thirty one morning because one of them couldn't get a document to print. Turned out the printer wasn't switched on. After that gem, and no apology, as it was obviously all my fault, I told them to find someone else, and now check who I do any work for before I start, and have a hefty call out charge (?100.00) for spurious and frankly stupid calls.

by inamhaque

My first unforgetable mistake was installing a new pc at one of my freinds home, that had 240V and 120V power supply switch at the back, the switch was set to 120v, which I didn't checked and plugged it into the mains of 240V....bugger I was so embbarresed....
Then recently I was creating a restricted user for windows NT on my work PC...but went into trouble creating one...so I though why not just revoke some folders right from administrator and allow the person to use it for a while....and then I revoked the rights from administrator profile and couldn't logged back in the machine...

Then guess I ended my night rescuing my files for which I didn't had backup and installing NT from scratch.

by leigh9

I was a workshop tech with a small ISP. I built myself a nice new machine under an agreement that it would be used to sell more of same. It was the first XP Athlon in the area and was fire engine red! I loaded the latest MS offering, XP Pro on it, no problem boss it's just the latest NT client. The moment I plugged in the ethernet the NT4 radius server and XP had a spat about who was boss around here and the NT radius server retired to sulk. Not only would it not talk to XP, but it wouldnt talk to the 800 odd w95 & w98 clients out there either....ooops! I learnt a lot that afternoon in a diverse body of fields, client communication, discreet apologising for beginners, and the guts of radius servers and ISP networks. Ouch!

by birketa

I had to go and beg the local DBA to restore from a backup when an application was discovered to terminate with extreme prejudice if a user called up a record with an apostrophe in it. This, however, was not the problem, my interim fix was:
update tbl set myfield="Great `O` Rings"

Because I forgot the Where clause I had that feeling of great doom as my SQL took more than a fraction of a second to run...

by deadly ernest

he said 'This is an open organisation speak up if you see any problems with this change in operations plan and budget'. Wrong

The whole new idea was put together by my boss an old time network design guy - his last network was coax. The Division Head, his boss, was a number cruncher with some basic IT knowledge. Neither new a thing about designing modern gateways.

The new plan was to lever a new partnership by selling dozens of new appliances (intergrated hardware and software that did routing, anti-virus and web proxy) to our existing clients. The intent was to just remove the existing gateways and put these in place when upgrading as they were cheaper than a the usual PCs with software, and quicker to put in place. Funding was included to replace some at critical points at out expense as we could remote manage them better.

Erroneously I believed the big boss' statement and told him that we should factor in a thousand or so for each install, to pay for a needs analysis and suitability report on the system as the new appliance was very useful IF the gateway was set up in a particular way - if not you had to redesign the gateway to suit the appliance as the unit was not adjustable in any way.

Neither undrstood gateways Big boss spoke with my boss, they spoke to marketing man from partner who made appliance - he had no IT knowledge in this area either. result I got laid off and a year later they both got no bonuses as the project ran over budget by 45%, over A$200,000. Seems only one system would work as a straight swap of the unit, the rest had major down time as they were urgently redesigned.

A coworker had kept a tape of the meeting where I spoke up, that got th bosses into trouble for not properly investigating my claims.

by gmotelso

Allowing the companies I worked for to label me as, and utilize me as an expert in a specific software package. As technology moves, companies rely on the expert to maintain the legacy application, often denying requests for training on new technology. When the company moved to new technology I was too expensive to retool or retrain. Younger talent was cheaper, and oftencame with the knowledge necessary to jump right in.

zzzfor years I was the person on the staff that kept things going. Now I'm a decade behind the times, with a lousy choice. Retool and retrain myself, and pray for another company to give me a chance to start at the bottom again, or change careers.

by michael.adel

I submitted a mainframe (you remember those?) print job on Friday evening just before leaving the office. In the shower on Monday morning I remembered ('remembered' isn't really strong enough to describe the lump in my throat) that I forgot to sort the input file by department.

You see, the report did a page break on department - so instead of having 55 lines on each page it created a new page every one or two lines.

When I got to work I walked into the computer room and told the operator to toss the twelve boxes of fan-fold easy-read paper that had my name on them.

I want to thank Joe (his real name) for never telling anyone about this rookie mistake.

by eric

Waaaaaaaaay back in college (1972), our college ran an IBM 360/48. (In those days, that was a mid-range machine worth about 1/4 million. With 48K (miniscule by today's stanmdards) of core memory, the CPU alone measured 6' x 3' x 8'.)

Late one night while, when all our batch processing was finished, I was experimenting IPL-ing (booting) the machine from different devices (card reader, tape drive). I was intrigued how the machine would respond when I tried to IPL from a disk that was not powered up.

I hit the IPL button. Within seconds, the machine shut down. All lights on the operator's panel were out except for one incredibly bright red Thermal Check light.

I opened up the access doors and found a reset switch. I hit it and the light went out. I then hit the main power button. So far, so good.

THIS time, I powered up all the disk drives and hit the IPL button. (No more screwing around - I was sweating bullets.) I crossed my fingers and hit the IPL button. Again, the power flipped back off and the Thermal Check light was the only light glowing were there chould have been about 100 happy lights blinking furiously away.

Gulp! I thought that was the end of my college career.

IBM was called and a crucial 8K unit of very expensive core memory had to be replaced. Instead of a nice array of neatly arranged ferrite cores, the Customer Engineer found a puddle of melted ferrite and tangle of burnt wires.

8K or core memory cost $24,000 --- 1972 dollars! (That has to be equivalent to something like $100,000 nowadays.)

IBM replaced the memory under warranty. (Let's face it, it shouldn't have happened. But then again, what kind of idiot deliberately IPLs that a machine that way?)

I was lucky. Had I not been doing a valid project on Operating Systems design, I might have ended up as a plumber*.

* Making $80,000 a year with steady employment! Maybe I should have been thrown out!!

by dsusman

I'm not sure which was worse.

Back in the 80s I worked at a Satellite campus of a University in California. There was no manual or training for using the comupters, my trainer told me (over the phone) to press each F-Key, then Shift-F, Alt-F, Control-F and write down what each one did.

One of those combination put our server into a looped diagnostic function effectively shutting down operations at our campus and one other. ITD spent a couple of days tracing the problem before they realized what had happened.

More recently, ITD had called to complain that our Department's server needed to be paired down, evidently 50 Gigs was too much to back up every day. I went into Groupwise to see what the defaults were to I could talk with the Boss about reducing some files. When I exited, I hit "RUN" not "CANCEL" and initiated a purge of 60+ data resulting in a 35 Gig reduction on the Server.

It took two weeks to merge the backed up data back into the Supervisor's mailboxes.

by guzmunth

Monday - Began a server consolidation project within a mid-sized national bank head office after months of planning.

Tuesday - Broke Nothing...

Wednesday - Server Farm Analysis - All good.

Thursday - Regional Bank Branch Logon Script Analysis. ?What?s this F:\ in the script??? There's no logical mapping to an F:\ drive anywhere - dump it".

Friday - "Eh, why are all 39 branches reporting off-line???" Restored script, reboot all regional servers.... 2 and a bit hours.

Saturday - Blinding Hang-over - angry wife.

Cost of project: $25,000
Cost of Alcohol: $50
People without cash for nearly 3 hours: ~800
Banking somewhere else - Priceless!

by maurice

Onced fixed a BT BACS system and collected 6 months of over duepayments.

Only problem is put a "Name" Jounalist over his overdraft limit.

Never got a thank you from The accounts department for recovinging 3/4 of a mill though.

by mill3502

I go back a few years in the industry and back in the good old days when everything was serial connections you did a lot of stripping of cable. Sometimes you got into the bad habit of stripping with your teeth. To make a long story short one day I was stripping wire and bite down good only to find out the wire was run along with 120 wiring and got a nice mouth full of 120v at 12amps. Never again used my teeth.

by speedracer94

In the process of upgrading a Novell 4.10 server, I wiped it out. It took an entire day to restore the server. The most embarrassing part was that I was in the midst of studying and taking exams to become a CNE. I took the final test 6 days after trashing the server. I didn't exactly trumpet that achievement to anybody on my team.

To this day, I have no idea what happened.

by tony.kenny

I was doing security cleanup work with a team of people. Anyways, One of the tasks that we had do was to delete unnecessary LOCAL accounts from member servers in a NT 4.0 domain. Well, one of the guys that I was with, logged into the NT 4.0 PDC, and thought he was looking at LOCAL machine accounts. He ended up deleting everyone's DOMAIN accounts.

Needless to say we ended up with egg on our face, and had a heck of a time trying to rebuild the Exchange Server.

by bigjimslade99

During a NT4 to W2K3 upgrade, I was browsing the NT4 domain and noticed about 2000 "tombstoned" computer accounts, wanting to cleanup the NT4 domain I figure lets delete them, on Friday afternoon no less. Well saturday morning my pager was going crazy with people getting errors about the domain unavailable. Opps. these were remote workers using laptops. These users would login with a cached account, start vpn then logoff and then attempt to login, no good. Took us a few hours to get the correct process to re-add them. 3 weeks later we had all the legimite accounts back and the real "tombstoned" accoutns removed. Nothing like scaring the crap out of yourself.....

by trmartin

I was still in college in the '70's, but was working part time - night shift as a computer operator on a rather large IBM 360 system. The console was a vast array of switches, dials, and lights, which overwhelmed, but intrigued this computer science major. I decided, for some reason unknown to me to this date, that the system needed an "IPL" (initial Program Load - or a mainframe reboot). So, I proceded with this task, thinking that I had the knowledge for I had seen this done once before by my supervisor. Well, needless to say, I screwed it up, but I did not tell anybody at the time. A few minutes later, when the system did not respond to console commands, I had to fess up. I was majorly embarrassed, but my supervisor was easy on me since it was a slow night. Those kind of things stick with you.

PS. After reading other posts, at least I didn't melt the memory. LOL

by db8abl

After developing a corporate wide system to evaluate projects and establish prioritizations my boss wanted me to train two younger staff on the process. The staff was trained and I was gone 6 months short of 20 years. The reward for hard work and dedication often comes too early and is given the wrong people. Over 50? Watch your back!

by ramyersfl

Had a client with an older desktop machine with lots of mortgage files. They did not back up, ever.

The PC kept locking up; random amounts of time. I fussed and fussed with it, loading service packs, etc. Finally, formatted and reloaded Windows but was only able to save a few things. The PC started locking up again after the reload. Turns out the fan on the CPU was stopping and starting at random intervals.

Someone else discovered this when they scrapped the machine. I should have looked closer at hardware; it is so easy ?to just blame Windows?. The client never paid me a dime.

by rbs4042

The biggest blunder I ever made was when I walked behind a telephone system and snapped the fiber optic cable off. We didn't notice it for about twenty minutes when we got a call that users were not able to make phone calls. Luckily we had a back up set and had it going in a few minutes.

Now the biggest one I ever saw was when my manager pressed the EPO button on the UPS to silence the alarm. EPO stands for Emergency Power Off and it shutdown the whole datacenter which took hours to bring back up. He swears to this day that he didn?t do it. About a thousand Lotus Notes users were down and about 300 desktops connected to the system. Not to mention the UNIX and Digital equipment that was providing the production systems information.

by james b.

Those of you that know Sun servers will love this. The 4600 series and similar models have CPU trays that attach to a centerplane. I was putting one in once and didn't notice it had a hard plastic dustcover on it's connector. the guidepins on the centerplane pierced this cover, and then promptly came out with it when I removed the board to see why it wasn't seating properly. Those centerplanes aren't cheap, but luckily they were able to fix this one. Thank god this was internal at a production facility, or I would have been in a world of hurt.

by ctrimbath

I have made quite a few "oops" in my time. I have learned from all of them and not made any twice. I think the biggest blunder I made was while designing and installing a complete new network for the company I worked for, I told my boss how easy it was going to be. I installed all the switches and routers (luckily parrallel to the existing network) once it was installed we started moving things to the new network to find out nothing worked. I forgot to turn on IP Routing......DOH!!!!

by rfishpool

A few years back whilst working for a large firm of housebuilders, the accounts managers pc was problematic.
Staying on after office hours I decided that the problem lay in a malfunctioning hard drive.
The pc was a two drive system with os on disk 1, and data on disk 2.
Didn't need to back up disk1, just reformatted & reinstalled os.
Disk 2 was not being worked on so decided not to bother backing it up.
Os booted fine, but to my horror data drive 2 had gone t**ts up for no apperent reason.
Never mind - will restore from accounts backup in AM.
Turned out most recent tapes they had taken were over 4 months old, and I had succeeded in wiping out 4 months worth of accounts info.
Lets just say i did not last very long after in their IT Dept, and am now a compulsive backer upperer!

by jwalsh

...."Keep it simple stupid"
I had a laptop with a black screen. After a few hours of dis-assembling the damn thing and putting it back together I still had no idea what was causing the problem. I then noticed a little slider switch on the side of the monitor and when I slid it up I had "solved" the problem.
I explained to the end user that the "video adapter had to be reset and the precision refractometer had to be recalibrated". He beleived that story, which proves my second theory "If you can't dazzle 'em with brillance then baffle them with bull#@%&".

by bassplayer and drummer

I didn't think much of what I was doing at the time. I was adding a Cisco Catalyst switch to two other switches in a high-availablity data center. This new switch was recycled from a call center and was already configured with its own VLAN database... and as a VTP Domain Server! I didn't clear the config on it, so when I connected it to one of the other switches, and set that connection as a trunk, the VTP Domain Server on the new switch contended with the VTP Domain Server on one of the other switches (you can only have one!). Since the switch I added had a higher rev. code, it won, and proceeded to erase all of the existing VLAN information on the other two switches, rendering every single port disabled. It was also the middle of a regular business day, so the effects were immediately felt.

It took me about 3 minutes to rectify this blunder once I was alerted. To my surprise I didn't get fired, but the director forwarded me an email from the CEO that read, "WHAT THE BLOODY HELL HAPPENED?" with his note added saying, "I don't ever want to receive emails like this again."

by mllwyd

In Unix, you can use the find command to also perform other commands. It saves a few keystrokes. I needed to find and delete several files. I carefully crafted my find command, specifying the directory to start with, the parameters of what to delete, etc. I was running it from the root directory, logged in as root. Pressed Enter and got suspicious when the command took more than a few seconds to run.

Killed the find and discovered it was deleting everything, starting in the /dev directory, the repository for all device drivers.

The system was still up, presumably because the drivers it was already using were loaded into memory. I wanted to restore /dev from tape, but the tape drive driver was gone. In a panic, I decided to reboot, totally forgetting that things like the console and keyboard also relied on [missing] drivers in /dev. Then everything was dead. I don't remember what I did to get the system back up, though it required me to be very clever and involved booting from the install disks.

After that, I started testing all my potentially dangerous commands in a sandbox.

by no bye-bye

As a fairly green mainframe field engineer, I was re-installing a system in a new facility. I got everything working except the A-D controller, and for the life of me I couldn't figure out why it wouldn't work. I finally swallowed my pride and called for help. They flew in the A-D guru from the coast, and after about fifteen minutes he found the problem: I had forgotten to connect a pair of cables under the raised floor. Fortunately the guru was sympathetic, and reported the fix as "repaired open circuit".

Another time I was troubleshooting a tough core memory problem. I had several double-layer memory boards stacked on the floor when I stepped back and crunched one of them under my heel -- $2800 each in 1970. I tagged it "bad from stock" and sent it back with my fingers crossed, but I never heard anything more about it.

I worked for another mainframe company that stocked customer sites with board testers, the idea being that we repaired circuit boards on site rather than swapping them out. A high-profile customer called on a Saturday and I went to fix the problem. I started testing likely candidates, but every board I tested showed errors. This went on for most of the day, one frustration after another. When the night engineer came in to help, he immediately saw that the corner of a plastic bag that a board came in was stuck in the tester socket, ensuring that every board tested would fail!

by Former member

It was 1972 and I was working as the lead night operator at a bank (before branches). Each day, four runs of checks and deposits were processed on a huge MICR reader. about every 20-50 documents a separator with a subtotal was inserted to aid in finding errors from unreadable checks.
After running the last run for the day, I asked the junior operator to verify that the totals balanced. When he confirmed that the amounts were in balance I gave the go ahead to begin sorting the paper checks by account number for filing.
He started sorting ALL the checks (which also stacked the separators away from the check). Only after the second pass of sorting did I discover that what balanced was only the last entry run; he had followed the beginning of the day process to clear all data entry files as he started the last run.
After thinking, I started reloading all the checks, which were, by now, one huge batch with no subtotals and with many of them shredded by the check reader. Fourteen hours later (at 8:00 AM) I had to explain the management why we had many thousand of dollars missing from the checks posted to customers accounts (we had already paid the checks) and no subtotals to identify which portions were out of balance.

by shublar

I had just went live with a large user provisioning project across all our major platforms. The Active Directory data did not sync properly with the central data store. I decided to clear the contents from central stores database for all AD platforms and then re-download.
Unfortunately, I forgot to stop communication to the remote platforms. I ran a simple SQL statement (Delete * from [datastore] where platform=AD). Dutifully and as designed, the system begain removing each of the 40,000 user and computer accounts from all AD domains.

Luckily, I realized the communication was live and stopped it just before it deleted the CIO's account...

by smurgymac

whilst working for a university architectural department as technical support, i was given a external harddrive by a member of staff,

diagnosed the problem as a distinct lack of partiton information and set off a standard data recovery program.

whilst the harddrive was being scanned and the program happily reporting recoverable data, i'd thought id leave it to finish scanning.

unfortunately the powersupply cable some how got tangled in with the chair leg and when i moved to stand, the chair tugged the cable and the drive proceeded to get flung floorwards, then hitting said floor with a sickening crack.

it looked ok but when i plugged the drive back in, all i could hear was a high pitched scaping sound and no reporting of any drive connected.

two years of documents that was on the disk were gone.

by rmackinnon

Early in my career, I was duped into working on a user's home machine. For some reason I got inside it and decided to pull out the CPU because I had never done that. So I put it back, turned it on, heard a pop and smelt burnt plastic. I had somehow put it in wrong! I was the idioit that broke the rule.

by dilberter

Managers have all of the bad traits of the worst
employees you have ever worked with. They lie and are egotistical, insane, and psychotic.
1. If a manager says hump out for me for two years and you will get a 25% pay increase--don't believe it--this manager will be transferred/promoted and the new manager will say
"That has nothing to do with me!?"--Advice--Get your money going in the door and leave the promises for the newbies.
2. Managers of one IT company wern't getting any new business in--sure enough the corp went belly up later--even if it took 10 years.
3. we backed up 1 database of 40,000 bytes and got 400,000 bytes--the manager said "load and use this database"--so we ignored his advice.
4. More later...as duty calls....

by lmassey55

I use a KVM switch to look at my servers from one console. I needed to install IIS on an old server to practice setting up SSL. I uninstalled and reinstalled IIS before I realized I was looking at the web server. Configuring everything to make the web site fully functional took several hours. I now have a graphic of each server's name in a HUGE font size as wallpaper on each of them.

by mizar-alcor

After more than 25 years in computers and networking, probably the one that sticks out the most is one day many years ago.

I was working to run a cable from the back of a Pr1me mini computer to the patch panel. The back of the cabinet was barely far enough from the wall to open the door, I was down on my stomach with my head and shoulders under the floor. (of course, the back of the cabinet was open, and the power supply switches are down in the bottom.)

You can see where this is going don't you? In the middle of the day, a simple task, and POW!, I nudged the main breaker to OFF when I rolled over to sit up. The sound of fans and equipment going silent was terrible!

Got it back on quickly, but of course, the damage is done. About a 15 minute outage in the middle of the day. Boy, was I embarrased, and humbled!.

J.

by errk'd guy

One day I was rattling around the switch and patch-panel racks
lamenting about my network administrators inability to use the
cableling system as it was designed. After running my monthly
diagnostic on the APC Backup-Power Tower, I decided to inspect
all of its connections on the back of the unit. Everything was
snug and I should have left it alone. But NOooo... I had to touch
the breaker reset lever. I barely touched it (I sware! :0) ) .. well it
tripped ... killed the power to the entire unit. Suddenly the sever
room was VERY quiet. I waited a minute or so, reset the unit and
everything came back to life.

Unfortunately, the database server had a seizure due to the
number of people who were accessing it during its improper
shutdown. After running the database repair routines we
reloaded ... but no go ... the biggest of the relationals was too
far gone for repair.

Now at this point most of you are saying "reach for your
backup". That we did .. unfortunately it was a week old. That
particular server had been rebuilt the previous week and the
reconnection to the nightly backup server had been missed after
the rebuild. I take the blame for that one to since I should have
double checked my guy's work.

Now the sinking nausea of panic begins to creep in ... but we
were saved by locating a recent ghost image of the troubled
database that was recent downloaded by one of our remote
offices a few days earlier. That reduced the missing data from 5
-7 business days down to around two days. Needless to say we
spent the next week re-entering missing data for our users as it
was located from other sources.

The next time you see a "Big Switch (any color)", ask yourself ..
"Do I really need to see what this does?" ... then walk away.

by theelkmechanic

I suppose it could be the time I formatted the boss's original floppy instead of the blank I was going to copy it to, but I'd have to give the nod to the time I was handling a tech support call, helping a user reinstall our application. This was back in the DOS days, and I didn't make absolutely sure what directory he was in when I had him do the del *.*. I had to spend an extra 45 minutes walking him through how the undelete command worked to get his root directory back.

by ahirsch

I was in the process of locking down a newly build server. I was flipping back and forth on the KVM a few too many times and thought I had gotten back to the intended new server.

While surveying the server, I noticed that it had IIS installed and thought that there was no reason to have IIS on the box as it was not a a web server. I proceeded to uninstall IIS.

A few minutes later we had a lineup of users asking why email was not working. I quickly realized what I had done.

I was able to get IIS re-installed and Exchange repaired in only 2 hours. Thankfully, it went without a hitch and the system administrator could only laugh at me.

by over-draft

Back in the 70's when I was a shiny new mainframe operator, I was working a weekend shift running jobs at a manufacturing plant. This was before any automated job submitter and the "schedule" consisted of a list of jobs written in a notebook with the appropriate input/output tapes. As the jobs were run, the operator was supposed to check them off. The schedule for the evening and weekends rarely changed and I'd been doing it for about 6 months so I was fairly comfortable with the routine.

I was running a series of jobs on a Sun. afternoon whose elapsed time was about 6 hours and whose outputs included thousands of pages of inventory reports that told various work units in the plant how much of what to produce in the coming week.

When I came in on Monday afternoon to work a swing shift my boss nearly had to fire me. Somehow I had skipped a job (even though I checked it off on the log) which resulted in not updating the files and consequently reprinting the previous weeks reports.

The most of the first shift at the plant (several hundred people) sat idle for most of the day while the recovery/rerun was performed.

I'm sure there were people further up the ladder who wanted me fired on the spot. But I think the operations manager really liked me and he understood that it was a manual process and simple human error is sometimes unavoidable.

I didn't lose my job over it, but I was kind of treated like a leper for a while after that.

by grundil

Ok, I've done all sorts of things, from disassembling / reassembling a laptop and having leftover screws to formating a hdd before backing it up. The best one though, that I've witnessed, but not participated in, occured while I worked as a tech at a small computer shop. The boss brought in a dual celeron machine from a friend of his for the store to sell. The lead tech was curious and want to see if he could overclock it a bit. His first attempt put it beyond it's capability and it simply wouldn't start up so he powered down and found what he though was the jumper to reset the bios. It was about 2 seconds after he powered it backup that we noticed a nice electrical smell, he hadn't jumpered the bios reset, instead he had bridged two pins on one of the motherboard fan jumpers, frying them all. Once reset the machine restarted fine, but none of the fans that were attached to the board worked, that entire set of circuitry was cooked.

by mreimers

Worst mistake I've made in my career has was when my Windows 2000 server with Exchange 2000 on it BSD'd after installing a printer driver. Called Microsoft and the tech walked me through steps. After a couple of hours the tech said to hard demote the server and then promote it again. I said you can't do that because the server was the exchange server and also the master FSMO. I asked him are you sure because everything I knew did not permit that process in windows 2000. Microsoft Tech said no problems and 6 hours later we used a backup to restore the server and 24 hours later the rest of the network was recovered. Backup! Backup! Backup! First... And if your gut says you can't do that get another opinion.

by mikef

We have, and yes, actual tears. We were upgrading our Exchange 5.0 server to 5.5 late one evening. Along the good ol upgrade path there was the part that wanted us to perform a backup, so of course we told it go ahead and backup. well, during the backup there was of course an error, so we call MS and explain the error to them and they say, oh, don't worry about that, go ahead and perform the upgrade. We did, and you guessed it, our mailboxes were no longer there, the users really cried more over the loss of their contacts rather than their mail, but they actually had tears streaming down their face. And BTW, we are not allowed, where we work, to back up exchange at all, never did and never will. Hopefully this is the biggest mistake I ever make during my IT career, because I don't think I'll survive another one like this!

by pete1978

It was in the days of DOS. I was at a site that was using DOS 6.21. All the PCs were suppose to be DOS 6.21. They had been delivered from the manufacturer as being DOS 6.21. It was a requirement to the manufacturer that all the PCs use a single version of the OS.

We had 1200+ workstations, so, whenever possible, we automated things. There was an update to a program that we needed to distribute. The update could be simply copied on top of the older version to work (that was one of the really nice things of DOS ... often much easier updates).

I took a base system, applied the update and then created an image using a program that compared files. The instructions given to the update script was to overwrite any older file with the newer file from the image.

Unfortunately, I left the DOS directory in the image and about 30 of the PCs were actually running DOS 6.20 (an older version). The update program could not rewrite the boot files, so the systems still booted in DOS 6.20 even though the DOS directory files were now DOS 6.21. When the update completed, the 6.20 PCs booted, but quickly generated incorrect DOS version errors as the DOS 6.21 files started to TRY to execute.

The real disaster for me was that one of the 6.20 PCs was on my boss' desk. My boss, of course, was not technical and had no sympathy for the fact that the manufacturer was suppose to put the same version of the OS on every machine. Needless to say my pant size was chewed down a few levels that morning.

Using the same update image software, I was able to correct the problem on all 30 machines in about 90 minutes, but my boss was still pretty upset that I had caused 30 machines to fail.

Oh well. The next update took much longer and when my boss asked for an explanation, I reminder her of the previous disaster. Enough was said ...

by catshev

My worst mistake was during a unix server failure. We had one of our production servers fail, and were trying to make rack room to work on the server, since our server room was crowded. I went to the UPS to unplug the failed server, and saw the label on the UPS at the plugs that said "Server1". I did not trace the power cord back to the actual server, and just pulled the plug. Of course, the label was old, and the cord was actually a different production server. When I unplugged it, the databases dropped hard, and did not come back up. We now had both of our production servers down on the same day!
Lesson learned: Trace that cord before unplugging ANYTHING!!!

by roywhelan

I was tasked with updating an old application that notifies our corporate office of IT outages. In order to save money and gain some Lotus Notes development knowledge I arranged a crash course on Notes Developement and tried to redesign the application myself.
My first challenge was to create an agent that would notify me when a field was not updated after 30 minutes. Some ameteur coding resulted in the agent emailing all of the corporate office every time it ran, as I had not specified how often or when the agent should run. The agent ran for 4 minutes before the phones started ringing, in that time we sent every user in our corporate office close to 1,000 emails.
: (

by marathoner

I was (a long time ago) working in R & D in a company that made an expansion sound card. They issued several prototypes of the card to the embedded software developers. If you wanted to listen to the sound they provided a jack in the back where you could plug in a telephone handset. The hardware engineers apparently had a lot of keyboard plugs laying around so they just wired up any old handy female looking thingy for the prototype jack, as that was not how the final product would be anyway. Of course this made it plug compatible with the keyboard.. I bet you can see where I'm going...

One day my boss was on vacation and I needed something off his computer. I turned it on without looking at it first and was quickly unnerved by that familiar burning electrical circuitry smell. You guessed it. Someone had unplugged his keyboard and plugged it into the jack in the back of the prototype card where the handset was supposed to go. Unlike a normal keyboard socket, that sucker was delivering 12 volts. Pfffft!

by bmedlock

Many years ago when I worked as a tech in a retail shop I was working on a computer for a lady who said her computer would reboot while she was in Word. I had been working on it for days and couldn't find anything wrong with it. She told me I had to be typing in Word for some time before the problem to occur, which I hadn't really taken the time to do, so I decided one day to sit down in front of it and start typing. I decided to be creative and wrote a cutsie little poem about how the lady says the computer doesn't work, but it works fine, and then I repeated a refrain that said something about defective end users, etc.
Appearantly when I closed Word I wasn't paying attention and just hit enter a couple of times and saved this document, which she found when she got the computer home. Luckily I was at lunch when she came in, and one of my co-workers caught me in the back before I went up front and told me to leave for awhile. I guess she yelled at my boss for a good half hour. When I came back he showed me what she had found, she printed it up and brought it in, and he told me, "Never do that again!"

by tuddekka

There are so many times I wish I could have done something like that. Defective End User. Trouble between Chair and Monitor. RTFM.

by s. daniels

After adding a secondary drive to our server, I thought I would be brilliant and I made the secondary drive "Active". WHOOPS! I lost everything on the server. Our office is a doctors office and we access charts through the server. Luckily the doctors could still get to their patient charts via the internet but it took me a couple of weeks to get everything up and running again. Not only am I the only IT employee for this practice, I run the transcription department as well so I was pulled in so many directions. I thought for sure I was going to be shredded by the doctors but I actully was told, "everybody makes mistakes". Glad I work with great people. People actually complimented me on screwing up because they said that things work so much better now and for not shirking responsibility. How could I anyway? I'm the only IT person in the building. Anyway that wouldn't be cool to blame anyone else. I biffed pure and simple.

Did I mention that there was absolutley nothing backed up? I now back-up everything religiously.

Now I am setting up a VPN. Hopefully it wont be another one of the above "learning experiences". Wish me luck.

by sboverie

The worst blunder was trusting a remote IT staff to guide me through adding a hard drive to a Banyan server. I was working as a field computer technician for a service company and was sent to one of our customer's sites to upgrade the server. The server had 2 SCSI hard drives and a backup tape unit, the hard drives were address 1 and 6 and the tape was 3. I was told that the company needed more drive space and to install a third drive. I spoke to their IT and told them how the drives were configured and that I did not know which order Banyan used the hard drives.

My philosophy in adding hardware is to make as few changes as possible and I chose to make the new drive address 4. I told the guy who was walking me through the upgrade and he told me that this configuration should work. HE had me format the new drive through one of Banyan's utilities figuring that this would speed things up. Basically, I was his eyes and hands on site and did what he told me to do.

After the drive was formatted I started the server and let it start up. It seemed to start correctly but some files were missing. The remote IT staff took over at that point and thanked me for installing the drive. They were thinking at that point that the server was working good but needed to move files around to accomodate the new hard drive.

The IT staff finally got the server running again but most of the data was lost. The people in the local office were rather hostile towards me for losing the data and causing their server to be down for longer than planned. They did accept that the faulty backup was their problem but would not believe that I was following the instructions by their own IT.

Hey, if the guys who walked me through the setup are reading this, I am sorry I was not more adamant about verifying the setup. That would have saved all of us the hours of grief.
Two hours later I was told to return onsite. It turns out that the drive I was told to format was actually the drive that the data was stored on. I was sent back to restore data from their back up tapes. That is when we discovered problem number 2, none of the backups were done correctly.

by ksm1192

Technical: Many years ago I had to insert a hardware performance monitoring probe into a live mainframe computer. I put the prob in the wrong place and brought the entire system down.

Non-techical: While I was presenting to the IT Leadership staff, which included my boss, I told him, he was completly wrong about something, which did not make him look good in front of his boss. He did not like that very much at all and pretty much ended my career at that company and eventually was laid-off causing an abundance of grief for me and my family for two years. I'm still paying for it still today.

by techmail2

Having sys admin responsibility for a number of UNIX boxes (in the days of the PDP-11), I was interrupted several times while doing cleanup work on various system directories.
After the last interruption, I did an "rm *" without verifying the directory I was in. As Murphy would have it, I was in /dev. UNIX doesn't take kindly to losing that bit of information ;-)
Fortunately for me, The 2AM backup tape was on the rack - and was good. All was back to normal in about 20 minutes. As we were a beta test site for some company software, the users just assumed it was another glitch in the "new stuff". I kept my mouth shut and the computer room techs wouldn't say anything negative about the only sys admin who didn't yell at them when they phoned for help at 3AM. It pays to be nice to other people, even in the middle of the night ;-)

by sdreelin

I'm a telecommunications guy and was a branch engineer for a company that sells digital phone systems. We had a customer who was a stock trader with about 50 to 60 employees call us with a strange problem with their phone system. I went in remotely via the system modem and checked a few things out and determined that a system reset would clear the problem. Unfortunatly at the end of the day when the stock exchange closes all the traders are making phone calls so the system was full of calls. We have a mechanism in the system that can busy out all the lines and as the users complete their calls it busies out the line. I watched the status screen until the last user dropped off and then performed a reset. I forgot that resetting the system also drops the remote modem connection so I was knocked out and now the system came back up without any lines working! I could not even call them to have them put me back in the system. I had to fax the user a message that a technician was on the way before they open in the morning! Luckily the tech arrived before they officially closed and all was well. Mind you I already had 6 years of experience with this system and just had a mind blank that day.

by oregonsteve

Well, how about taking down the entire Server room: Unix, Mainframe, and Windows!!! We had just installed a new behemoth UPS. "Wow!" What a wonder it was. Now we could feel confident that in the event of a power failure, we'd have a whole 25 minutes to arrive on site and down everything properly to prevent data loss.

But what would happen if the UPS went down? No one may have been asking themselves this question, but we all got to find out anyways, courtesy of yours truly.

On one of my routine morning checks of the servers, following the installation of the UPS, I noticed a big red light on it.

Well, in my experience, a red light means that something's wrong. I stepped closer to the unit, and reached out my hand to touch the light ever so slightly... WRONG ANSWER!!!!!! It wasn't just a light, it was a button. And it had these three letters on it: EPO For those of you unfamiliar with this acronym, as I was, it stands for "Emergency Power Off". And that's just what happened. Power for the entire room was immediately terminated. I gulped hard, quoted Butch Cassidy as he's jumping off of the cliff, and then called my manager. "Uh, I think we need some help down here. I just killed the power in the server room."

Well, believe it or not I did NOT get fired (even though if I were my manager I would have drop-kicked me right out the door!).

After that the Facilities department made a little plexiglas cover for the button save others from the terrible fate if they were also ever inclined to press the BIG RED LIGHT!!!

by Member 26053643

I work at a small private university and was also attending classes at the time to finish my degree. I got my first ID to delete and popped right in and deleted it - only it was the wrong one. The ID I deleted was that of one of my professors that semester for a programming class. I did away with all his files and emails - and the man was working on his Ph.D.! I nearly died! We were able to recover his files, but not his emails. He was exceedingly gracious about the situation. And no, I didn't fail his class. In fact, I got an A!

by sansevieri

Back on 1998.
Load a printer driver on NT 4.0 was the challenge specially on the production prinetr server ooppps.
The problem was It was a sunday an I thoght i was alone on the firm.
(I forgot that all the lawyers from the firm were in a meeting on the conference room)
Needless to say Lawyers like to print a lot.

by the igneous group

I used to manage several VAX 11/780's These had a wire-wrapped back plane that was custom created for each system. On a system upgrade one of my peers from another state had to put another wire on the backplane for a "jumper" due to a new hardware upgrade. I let him.
He put the wire on the wrong post and fried the backplane, melting about a dozen wires. Frying several of the cards in the system. I had to re-wire wrap the backplane and coax the system back to life! Needless to say - I was lucky to keep my job. (Don't get me started about the other guy!)

by Member 25348569

went so well that IT Admin and I (an old friend) went over the pub - luckily, I had just one beer - before returning to finish the job.

To make some minor amendments without mounting all of the disks, I brought the server up "-NA" which, in earlier NetWare versions prompted for a server name, which could be anything as it's lost on reboot.

NW4 does the same - nearly!.

The IT Admin was p:ssed and was annoying me so when the "Enter Server Name" prompt came up I type FUCK_OFF_YOU_BASTARD to give him a laugh, edited the config and restarted the server. It all tested OK, came up "MERIDIAN_01" and all of the IT container were able to log in. Alas, we were all in the Server Context...

"OK, everyone. You can log in now"

Login: Mavisp
Cannot find server object FUCK_OFF_YOU_BASTARD
F:>

I'm in London, so after troubleshooting for a while (after I've hidden for an hour, calmed down and overdosed on tranks), I called Novell Tech in Germany but he couldn't seem to grasp what was going on. I called Provo and told my sorry tale and I thought that the Tech was going to explode. When he'd told everyone in the building and they'd all had a good laugh, he gave me the fix to remove ghost objects.

I asked the Tech that if I'd have typed PISS_OFF_YOU_BASTARD, would anyone ever have known as the server names are checked alphabetically. I'd have been safe.

Moral. always expect the unexpected.

by tnunetworksupport

We had received our new APC Symmetra UPS within the past 1-2 months. I was configuring the APC PowerChute Plus software on our servers so the servers would be shutdown automatically when the UPS switched to DC for longer than fifteen minutes.

On the configuration page for the client software was the [checked by default] option "Turn off the UPS after the shutdown finishes." I didn't read the documentation and did not catch this. When I tested the server configuration, by manually putting the UPS on battery, the server shutdown as expected. Then, the UPS shutdown also ... taking our entire server room (servers, core network equipment, telephone system, etc.) with it! Needless to say, I felt pretty stoopid!

:)

by oisleach9

I changed the application base classes used in a range of different products, and while I checked the building etc. I forgot to check the operation of the various applications and as a result, brought 5 product lines to a complete halt.
After 1/2 day of running around I was given a roundly sarcastic talking to and then told how to fix the situation. After 8 months on the job it was an eye opening experience so that I have slowed down everything I do.
So now I get complaints about how long everything I do takes.
Damned if I am not careful, Damned if I am careful

by marathoner

I had just started a new job. I was tasked to port this huge database and deploy an isolated test copy on a private box so I could beat up on "realistic" data. There was the master and the slave. Obviously you dump the slave, because a database throws up a write lock while it's dumping and you don't want to dump a live database in the middle of the day with several hundred users connected to it. You can dump the slave, because write throughs to it will just be deferred until you're done.

Did I mention this was a HUGE database with several million records which took more than a trivial amount of time to dump itself especially if you're piping it directly to the network. :P Well apparently the site admin had been a little inconsistent with naming the servers, and he had very strong opinions on what their names should be, which conflicted with what the boss actually named them. As you know if you run linux, server names can come from one of several places and at that time there was nothing to assure that the names were consistent. Being brand new, I believed the server names I was told and didn't check against IP numbers, and inadvertently started dumping the master at 10:00 in the morning. Boy did my phone ever ring!
I had just disconnected about 600 remote users in every state in the USA and a few foreign countries.