What's the worst mistake you ever made as an IT pro?
This is page 2 of the discussion. Read the opening post
After Hours
This is page 2 of the discussion. Read the opening post
by thefraz
The first was when I start "officially" as a member if the network administrative staff. I accidently moved the complete email user list into another folder area. Took a while to figure out what had happened and some more time to "fix" the mistake.
Second time occurred when I installed my very first "high-speed" CPU onto an existing motherboard. Thought I'd save the client some money and use an existing CPU cooling fan. Wrong! It took about 15 seconds to fry the CPU. An expensive mistake but also a learning experience.
by paul.tiffany
Some years ago, the company I was working for got a new client. One of our tasks was revamping the data structures to align with the business needs. One of these needs was to print names and addresses for large volumes of mailings. The name field had been defined as 85 characters and we decided to truncate it at 33 after careful inspection. We started a process to run over night to change the fields and structures.
The next day we got a call that all the names were now three characters long. No problem, we had backup tapes going back for over a year. When we tried to reinstall yesterday's database, we discovered that the backup process was only saving the operating system - no data.
You can imagine the nightmare it took to reconstruct the database while our client was essentially out of business for over a week. Dropping one character (in combination with other errors) cost millions.
by rbs4042
by tonythetiger
I once thought I was mistaken about something but it turned out I wasn't :)
Seriously two stand out. I once rolled out a (THE, at the time) server to get the numbers off the back, and miscalculated the length of the power cable...
I have also dropped a brand new 21 inch monitor. Very loud!
by bg6638
by ibanezoo
by dark_15
Probably my worst mistake was when I setup a few backups jobs. A program we use at work normally makes a backup file of all its transactions just in case something were to go wrong. The manual itself stated to backup these files instead of the main data files. Not to mention, the backup files are 1/10th the size of the regular files - which set of files do ya think I'm gonna backup?
Well anyways, I got a call saying that a database got corrupted. So I go to the backup tape, copy the backup file, and then revive the database. I give a go ahead to try it out.
Well, I get a call a few minutes later stating the file is 4 months old... and is not the recent revision! I double check the tape, and I had backed up the backup files correctly...
So I make a few inquiries and it turned out my boss decided to shut off the auto-backup feature of this software... and since I had backed up only the backup version of the database... well, let's just say my boss wasn't too happy.
Lesson learned: backup everything, and not just what the manual says...
by mickster269
(Ok, I knew them already, but just in case):
1. Make backups daily.
2. Don't touch the Red Button.
3. Make backups daily.
4. Make sure the backups WORK.
5. Spell check your commands, before you hit the "enter" key.
6. Make backups ...daily.
7. DON'T EVER TOUCH THE RED BUTTON.
by mat hancock
by manuel.amaro
Pay attention and give the benefit of doubt to end users.
Some time ago I was very tired of a long day of users phone calls (working on a help desk)(I think it's hell's closest area) and I got a call from this user that can't login.
I asked him what was his username and he answered me "ogemel" and I said, "I'm sorry, that should be your password, you shouldn't reveal it to no one, please tell me your username, it should be similar to your name, what's your name, please" and he answered again "ogemel", then I gone beserk and asked "Can you hear me? I dont need to know your password, tell me your name, where are you calling from?" and then my coleague at my side, called me, "Hey, be cool, we have a user called Holger Melo, and he has a funny accent".
My face turned red and I had to disgise my stupid attitude with all education I could find.
by ericl_w199
was plugging in a new server and apparently was too much power for that rack.brough down 10 servers.mostly web servers.but it also had our main dc and exchange....yea that produced some calls.my manager found it funny.?.?
put it wrong date for a job and made the system charge everyone single account an overdraft fee.fix.....manually reverse all 30,000 accounts or pay 25,000 to the company that provides the server and software to fix it.didnt get in trouble for that either.
by dr_zinj
and inserted my foot to the hip.
While working for the military, had a subordinate who would take off for various reasons, frequently not leaving notice of where he was going or when he was coming back.
His wife had a habit of frequently calling him at work.
She caught me on the phone at the wrong time and I just blurted out in frustration that ,"I don't know where the hell he's f*cking off at right now."
Manager damn near canned me on that one.
by gsg
by vzicomputer
by bndplus2
Becoming one.
Seriously.
I work for a company that has no concept of the NEED for training or the need for the proper tools.
Lack of management support and vision stops me from being able to do my job.
I've been in the same place for 11 years, doing the same job. Increased responsibility (with no authority, I might add) as well as tasks, with nowhere near the same levels reflected in promotions/raises. Management second guesses me because they read something in Newsweek and thinks it applies to us.
I'm also tired of being the sole Admin around here. I have a pager, a CrackBerry and another cell phone. I work nights, weekends, vacations...
Problem is, I make enough now that leaving would mean a pay cut - at least for a bit. That makes it difficult. And since I don't have a Bachelors, that ties me down even more.
So I'm working on some certs. Certs are BS, I know: it's a test that gives you a piece of paper. Cert mills have ruined their value. But if you don't have one, you don't even get looked at.
Hopefully, in a year, I'll be working somewhere else for a more reasonable employer.
If I had it to do all over again, I'd have chosen a different field. Possibly not even computer related ("talent" is getting cheaper and cheaper, so why work with computers?). Or, perhaps, my employer is the issue. Hard to tell, as this is the only purely IT job I've ever had.
Anyway, you asked. I (kinda) answered...
by bg6638
by hgdyahoo
Was working on major changes to a customer's inventory structure, and had a long SQL UPDATE to add leading digits to their part number. I got as far as "UPDATE INVENTORY SET PART-NUMBER=" and bumped the key with my notepad that contained the rest of the statement I was typing in. Since it was a server-side request, I couldn't stop it as it went through the db wiping out all of the customer's inventory.
THANK HEAVEN I had sense enough to have made a full backup and lock the db exclusive BEFORE I bumped the key. It took 10 minutes to restore, but I was sure glad I had the Backup!
by conundrum
The biggest mistake I made was to suggest to someone to use Explorer to locate her restored files that I had put in a subdirectory. This user believed that files on a server could accessed by anyone. One day I had to update the OS on her computer and saved her files onto the server, and restored them to a different subdirectory and told her where they were.
She was so upset about my suggestion that she had the director write a letter to my boss complaining about my people skills that I was transferred to another office after that.
I learnt to handle users better after that.
by thumper1
Back in the days when we were using systems with the old AT power supplies, I had to replace one that was about three months old on a system owned by the administrator of a Medical center. It had about 350 meg of data, (DOS) none of which was backed up. After I plugged in the new power supply I turned the system on. My first thought was where the siren sound was coming from. When I realized it was the CPU fan running at about 10,000 RPM, it was too late. Various caps on the system board started exploding. In short order the room was filled full of "magic smoke" (All electronics depend on magic smoke to operate. If you let the magic smoke out, the said electronic device no longer works) All this happened before I could get the switch on the power supply turned off. Upon removing the power supply and testing it with a volt meter, I found the 12 volt circuits really had around 36 volts and the 5 volt lines had were in the 18 volt range. We obviously had a new. defective power supply. Unfortunately it took pretty much every component out of that system, including the hard drive. The customer was livid, the fact that I this was the first and only time I have ever had this happen made no difference to him.
From then on, we tested every replacement power supply when it was unpacked.
by dsfromca
by jeff.allen
We here in Oz get power supplies sometimes set to 110Volts. They can make a pretty display!
I was installing such a beast (no external switch to set input voltage: it was by internal link), in a Unix server some years ago, in front of an anxious customer. Yes, it went bang (very loud!). His calm and collected response? "It's not supposed to do that, is it?"
by thumper1
by mecarlson
by napmax
by manuel.amaro
In my IT beggining I was reinstalling our finantial application (MSDOS back on 1988) and I thought I've made a backup of the data directory and I've erased it. Thinking better (but too late) I've managed to find the place where the backup was and... It wasn't.
5 years of Company's Finantial just vanished under my hands (our backup policy sucked), I was already feeling myself unemployed.
Then I called (in secret) our consultor and he introduced me to Norton Utilities and we've recovered those %&$#&%$ files one by one. In those days I thought he was some kind of miracle maker.
by manuel.amaro
Some time ago I've created a batch file called f$#$.bat that answer "F$#% YOU!!!" when you write the word on the command line. That worked when I fell frustrated with something.
After some time I've found one of my coleagues completely crazy, running all kinds of anti-viruses and anti-spyware because he was mad with his programming and wrote f$%#& and the computer answered back.
We got good laughs when I explained him
by techierob
This still haunts me to this day. Every year we have a main sales day where we put all of our stock (I work in the vehicle industry before you ask) on special and we have a massive fuss over the whole deal.
Anyway, I was in my first year and was just running around making sure everything was working and in order. Only one department was using their computers, so I was asked to keep an eye on the rest of the "technical" stuff. I noticed that the pathway between our comms room and the lounge (or lunch) room was covered in limestone footprints and looked pretty awful. Being the conciderate type I decided to grab the vacuum cleaner and clean it. Being that there was no power outlets near the corridor, I got an extension cord and plugged it into a spare outlet on the comms room wall. Little did I know that all the outlets ran off the same curcuit and sure enough, the trusty vacuum cleaner overloaded the circuit and threw everything on it out!!
The UPS's got locked in a power overload state and refused to power on - the main circuit breaker blew and I had siumltaneoulsy knocked out all of the phone system and computer network in one hit!
Half an hour later and absolutely soaked in persperation I managed to get everything back online. Now whenever there is a network slowdown a manager always makes the comment "Not using the bloody vaccuum again are you??"
Gotta love that :)
by rcetlin
At our office, the common response to "I have no email!" is, "Do you still have a job?"
Most lower level folks get a kick out of it... Higher ups unfortunately do not have such a sense of humor.
by kblume18
by burned out
About 5 years ago someone gave me a motherboard of a printer to troubleshoot some minor issue. I put that motherboard in a shopping bag and hanged that bag on my bike's handle. On my way back to my office, when I took a sharp turn, that motherboard got stuck between bike?s handle and oil tank and broke into two pieces :(
I spent whole night in joining the broken electrical contacts but that board still did not work and I had to pay the cost to its owner.
I used to work for a small company whose customers used to short invoive (actually, short the payment) to get the owner's attention so he would call and actually talk to them about some problem they were having.
My mistake? He asked what he should do about "this short invoice" and I, fresh from a stint at a collections law firm, suggested he call them and see if they could collect the invoice ... while the customer was on speakerphone.
On the plus side, he never asked me that stupid question ever again, and at least one customer never shorted an invoice.
Let myself get hired by this place because their customers were buying PC's instead of expensive terminals for their UNIX servers, and they needed a windows guy.
...
by russmiller88
My buddy and I were the 'computer experts' as collateral duties at Navigation school. Big extensive systems provided by the USAF for flight navigation training. Big RED button on the panel, covered by a clear, plastic flip-up cover. EMERGENCY. That sort of thing. We ran a LAN, had worked on computers since the 286 and earlier. What does Ed do?
Pop the cover. Push the button. Guess what? Everything shuts down very quickly and lots of civilian technicians start setting their hair on fire.
Lesson learned? Don't let Marines near your Air Force computer simulator 'cause we're gonna push every button...
New server, upgrade OS, upgrade Eschange.
Except the owner didn't have time for "that project management nonsense" and his wife, ever the Cardassian-style "leader", thought her every modification to the scope would somehow reduce the contracted price. This included, but was not limited to, upgrading her OS, upgrading her version of Peachtree, reapir of an aging workstation, and adding it to the network.
We should have known immediately after the contract was signed, and the initial payment was called for, and the owner said, "well that's what we agreed to, but you didn't really think we were going to give you an upfront payment, did you?"
by baketown83
We had this file server that serviced over 1000 users. Well, I was trying to fix one users permissions and by accident deleted all permissions to a few of the most important files on the server.
Luckily I had some contacts that I new and I called them, let them now what I had did. After a quick laughing session they quickly educated me on my mistakes and how to fix them. Also to my advantage this was during the graveyard shift. A few managers were a little pertubed but thankfully I resolved the issue before the big "cheese" showed up. But I can say that I gained a ton of knowledge about permissions.
Sometimes I find that I have to make mistakes in order to learn. How about you?
In my first programming job, I was maintenance programmer for a major steel company's hourly payroll (about 250,000 workers' checks"). It had a habit of blowing up about every other pay run, in which case the programmers had to fix bugs in time for the money to reach the mills the next day. If it didn't, the workers had the right to walk out at 12:01 PM -- while management walked all over the programmers' heads.
Note that source code was kept in boxes of punched cards -- about 2000 per box -- of COBOL code. Absolutely necessary for rapid recompile and link.
So of course, one lonely Thursday afternoon before payday, yours truly pulled out the Gross to Net program deck, and began making some minor changes without first duplicating the deck. I set the box on a nearby wastebasket, which was already half full of miscellaneous card waste.
Sure enough, 5 minutes later I bumped the box into that wastebasket, irretrievably fouling the production source deck. There was NO backup. Additionally, all the (very fast) keypunchers had left for the night -- there would be no help from that quarter, either.
What ensued was 6 hours of frantic punching AND verifying of a complete listing for that program, followed by a card printout and sight verification which took the rest of the night. Not only that -- we were (unbelievably!) paid overtime in those days, but I couldn't TELL anyone! I could only hope that no bugs developed that would require maintenance to the code -- because it would have been fatal hours before anything could've been fixed.
Thankfully, no bugs showed up that night, and the deck was put back into service with no one the wiser. There were some suspicious looks directed toward my bleary eyes the next morning, but at least I HAD a job to go to -- much better than the alternative!
by dukhalion
by tom_in_nj
by crimper
by wrstead
Years ago I was working as a system integrator (read outside tech. consultant) and was contracted to an insurance agency to roll out a new e-mail system for their office. I had a good working relationship with the office and was well trusted by their management folk.
I arrived mid-morning with the required equipment and their Accountant /?CIO? told me where to make some room for the new server case and then left me to work my magic. The new mail server was fairly well pre-configured so all I really needed to do was "nudge" one of the older servers over a little, connect the new mail server appropriately, tweak some final settings in the mail software and I was off to the desktops.
The old server was heavy and needed a little wrestling to slide across the floor and just as I reached around the back to get a good grip I heard the sound level in the room change. I had just turned off their Agency Management database server while a couple dozen people were logged in and working.
I had no choice but to sheepishly seek out the woman who had just entrusted me with her server room and explain why her phone was about to start ringing.
The database server was well-configured and very little data was actually ?lost? but I had the email system up and running for hours before the mirrored set had finished its repair process.
by johnnysacks
I'll never forget that feeling of doom...
Thank god the table had a couple million rows and I had time to yank the network cable out of the workstation then kill the process from the admin app. Took a while to roll back but I had to go for a walk outside and contemplate a career change.
by praveen
It was on the early days of my first job.
I removed the computer from the windows 2003 domain and I do not know the local user id/password information. That computer had lot of valuable information and I was really panicked.
Finally I found some online tools which could log on to the computer without a user/password which saved me.
by cswearingen
Maybe I shouldn't fess up to these but here goes:
1. While in college I interviewed for a PC Technician position. The IT Director knew I was going for a IS degree and asked what my least favorite programming language was. I told him I hated COBOL. Naturally I found out, after opening my mouth, that they were a COBOL shop. But I still got the job and upon graduation was promoted to programmer (I've since moved them from COBOL to VB.net and C#).
2. Early in my career I accidentally disconnected the external SCSI backup drive for a Novell file server. Without thinking I immediately reconnected it (while it was still plugged in to the wall). Fried the server's MB and the tape drive.
by panzrwagn
by cswearingen
Just the other day a close friend told me she had gone to Best Buy to purchase a new scanner. She's a teacher and wanted a scanner that had OCR software. Upon describing what she wanted the knowlegable sales guy AND the "Geek Squad" guy BOTH told her that the technology didn't exist.
She bought a scanner anyway and found the OCR software in the box when she got home. So much for knowldgeable staff and consultants.
by sauna
by sauna
by tzutzasialbu
Your story sent me back to one of the worst times on my job...
The UPS was loaded with 8 switches and around ten servers (Big mistake). It was going like that for almost five years. One sunny day, the UPS gave me a warning that the battery test failed. I thought that it still going to work for a few weeks (Another big mistake) and I was going to look into it at a later time. Couple of hours later, the UPS shuts down and with it all my devices. My luck was that this happened on a Friday at 4 p.m. and it didn't impact the production very much. It took me a whole weekend
to make the network functional again.
by 3xp3rt
After 15 years of experience in IT someone come to me with an old 586 machine.
After disassembling and cleaning (I never see a computer with so much dust inside) I begin to assembling back all the components. I forget that this type of CPU you can plug in even in wrong position, so I did it. After power on of course a lot of smoke and one burned CPU.
by wfs1946
My biggest excuse of a mistake was sending back 3 brand new computers because I couldn't turn them off. It seems that a 5 second timer was added to powering off (you know, you have to hold the button in for 5 sec) and somehow in all my reading I had missed that new feature.
Three new PCs and one vendor's course on BIOS settings and everything was fine.
by pierre!
the way that the power worked. They went from "power button kills all power" to "system board still live with power off" mode.... and it was time to upgrade ram on the production web site...
I am *now* the ONE you want to upgrade ram... After frying the dual processor mobo, I was able to locate one in California and have it FedExed in the next day.
Till then, the Web Site was down for the firm I work for... and I now unplug every PC I work on before opening it up. I am also religious about the 3M static wrist band...
Older and wiser I am...
by beilstwh
by p_piluk
by nusigf
Windows 3.1 was perfectly fine. Why, oh why, did I ever look at the shiny blue box with the 4 color logo? Like a lemming, I followed the trail right to Bill's pockets, inserted enough money to feed me for a month in college, and hurredly made the trip back to my dorm room. I quickly whipped out and installed... junk.
Luckily, I still had my 6 Windows 3.1 disks, 3 MS-DOS 6.22 disks and ran a FORMAT c: on my 60MB harddrive, edited my autoexec.bat and config.sys files to optimize the 4MB of RAM and continued to play "Scortched Earth" with my room mate.
by msg2612
I once accidentally deleted a public folder on our Exchange Server which held over 1200 contacts - no backup.
Luckily, I was able to extract the data from an old .edb file.
by jgmsys9
by fregeus
by creative8008
by kashif
by tommy higbee
Accepting a change in my job status to "Salary - Exempt" with a pay raise. Never really had overtime to speak of before that. But ever since I no longer get paid for it....
My favorite goof was someone else's. (Naturally.) At the small hospital where I was working, we had put in a new UPS and everything was on it. It was a nice big unit, almost chest height. To power it off, you had to push and hold two buttons about an inch apart, just to make it hard to shut off by accident. Well, our cable guy had to reach some devices that were high up in the computer room, so he stood on the conveniently located UPS. Turned out his foot was just the right size to hold in both buttons at the same time.....
It's probably good to make these dumb mistakes, so we're a little more patient with the person who calls us in at 5 in the morning because a crucial printer isn't woking, and you fix it by ... pushing the power cord in a little further in the back of the unit.
by scottyc2005
by runcie
At last we had internal email. I had just figured out how to do MSAccess 1.0 stuff. I wrote a long document, plus screen shots, covering my discoveries and sent it out.
"What are you doing that is filling up our storage?"
"I sent an email to a heap of people, but that is all."
"How long was the attachment?"
"Why would that matter? Surely Outlook just points everyone to the same copy?"
(I was a poor innocent soul back then.)
"Well, as soon as we free up memory your email's attachment is filling it!!"
You guessed it, I had sent the email to everyone using MSAccess at our place... we had rather a lot of them.
by runcie
Something was troubling the hardware engineer. (Unknown to me, the alarm system in the main control room was spewing out masses of alarms, serious alarms, like "radiation counts souring sky-high".)
"Can I watch you work for a while? We think you are the common factor in a situation we are analysing."
I could not see how, but not wishing to be unfriendly I said, "OK, be my guest."
It was late at night (1968) and I was testing a burst-can detection program in a nuclear power station. The paper tape shot through the reader at 1000 chars per second. It generated heaps of static in the plastic holder. I amused myself by seeing how long the flash of static to my pencil could get. I got about 1 inch that night.
He leapt into the air, fully awake.
"Eureka, that explains all those alarms!!!"
Anyway, a few weeks later the false ceiling caved in. It was due to accumulations of dust from holes in the concrete for rock-bolts to hold services such as heating pipes being added way above the computer room.
The cold air and dust did massive damage. These same engineers then had to replace heaps of relays, and our drum storage (pre-disc days).
I didn't feel so bad after that!!
by quintar51
This was one of those days I was going to call in sick, but decided to be a 'trooper' and come in anyways...
Part of the image process is to run a backup script, which I just forgot to run and blew the system away.
There was no backups or anything.
Lucky for me this user wasn't a big fish or else I'd be canned for sure. The worst is that feeling in the back of your throat when you're about to tell the user that all his files from the last 2 years are gone....
by raines
When I was a very new network admin, I needed to work on a printer problem. The printer was right next to the server and the server UPS. I was leaning to try to get a better angle on the printer, and heard a "click" down near my foot. I had just powered off the server. I brought things back up, apologized to the users who had lost their work in progress, and then I built a little cardboard protector around the power switch.
People were pretty good about it, although I did get some ribbing for my "big feet"....
by icubub
by mcguire.tim
Back when I was first a tech in the 1980s, I worked on Apple IIs. One thing I knew as a newbie tech was that chips on the motherboard had a notch on their tops to show you what direction to place the chips. All the chips notches should be on the same side, I knew this if little else.
Always there is an exception. Apple IIs where the exception. Their processor could be upgraded. Many people did this. When taking out a chip you did'nt look at the notch. When putting it in you did.
I lined up the notch on folks apple IIs being very careful that all the feet where in all the proper holes.
I would turn on the machine to test the new processor and they would immediately burst into flames, well, they would smoke a little first.
For some reason the notch should be on the other side but I burned three mother boards learning this little nugget.
by bertagirl
by rexworld
This was back in the day, I'm thinking maybe 1998 or thereabouts. I was working as an ad support engineer here at CNET, and one of the tasks was to periodically go into the directory on the docroot where we stored the ad gifs, delete them all, and then rsync the latest set of gifs from the ad server.
For reasons I don't fully understand, instead of typing "rm *" I did "rm /*". Essentailly I deleted the entire disk drive containing our docroot.
Luckily of course it turns out in Unix (and I think most operating systems) a delete like this doesn't really delete anything except the link to the directory. Our senior unix admin, Ken Emery, was able to restore the docroot with a couple keystrokes.
But man was I in a panic there for a few minutes. It's one of the reasons CNET instituted a policy where only Ops engineers have the logins to live servers. Us application engineers have to file deployment requests thru Ops to get stuff pushed out to live.
by lock'emdown
by gurdjieff
by ssyoung1441
by ijusth1
by ijusth1
by critch
Sitting in my office, I could look through a glass wall to our bullpen, through another glass wall to the'puter rom and through the far side to the hallway. I see the HR manager with some other guy at the far end, nd idlely wonder what that wand thing is. To my dismay, the HR guy pulls out a swipe & into the computer room they go. All of a sudden I realize IT IS A SMOKE WAND. The numskulls were testing the firealarms with a smoke wand. I ran & vaulted a desk & tried to yell, but yes, the alarm worked & just like it was supposed, it killed all power to the room... the mainframe, the Unix servers, the Novell servers, the AS400 and the telephone switch.
I took the card away from him.
by quikvpn
Back when I was new at using GHOST I had a client who wanted to upgrade a 20 GB HDD to a 40 GB HDD. Long story short I ended up GHOSTing the blank 40 GB drive over the full 20 GB drive. Thus I began living by the motto: "Before you move forward make sure you can go backwards."
The client I had was pretty cool about the whole thing. He had actually backup but his Quickbook databases before I hammered his drive. The reset of the data was stuff he said he could live without.
by quikvpn
by pwalters9
by jeremiah2911
See, you have to be able to think of a good excuse on your feet.
I once did a similar thing I asked another tech if we had any spare power supplies after my [faulty] diagnosis. He went to get one, and I noticed the thing wasn't plugged in. I plugged it in and tried the PC with power. It powered up just fine. Just then the tech came back with the PSU. Thinking quickly, I grabbed a spare power cord that was lying on my bench and threw it away. "Nevermind," I said, "it was a bad power cable." After he left for the day I retrieved the discarded "bad" cable.
by nerdy_gurl
I'm the general IT person for a very small company and a couple years ago we decide to upgrade to a couple new servers and Server 2003. We chose SBS2003 for the DC (we were running NT4) and Win2003 server for the file server. Most of these final decisions were made by the boss's bro who has a PHD in comp sci and has had an attitude about me since he came onboard (I'm self taught in computers). He is notoriously cheap and thought that saving some bucks by getting the new SBS2003 in the Dell bundle would be a good idea.
I'm careful about backing up and CYA whenever I upgrade anything so I read a lot about the new 2003 OS's, and went to a couple of free workshops. Because we're small we don't have a lab for testing things like this, although I often will on my home lan which runs Win 2000 Server. But I just didn't want to put an eval copy of 2003 on my home lan because then I'd have to replace it later with all its attendant crap. I thought I'd cut it OK since I had no problem installing NT at work and Win 2000 server at home.
We buy the Dell hardware with the preinstalled OS's. The file server setup went fine; we weren't using AD with that one (Those files were on our original NT box which was our PDC as well.)
Now the DC upgrade had to be scheduled for when nobody was using it (which was just about never!). I was going to do it while folks were away at shows since that was the least busy time, even weekends were bad, and show times had 4 days with little network use. So a show period rolled around and I began. I had decided to do the entire change from scratch including all accounts since the NT was so old. Also I wanted fall back in case things went wrong; I could go back to the old server with its account information serving the network.
So... a long line of problems started, separated by long periods of waiting for a 'not very busy' window to finish the job...
First, Disk 3 of SBS2003 was buggy (taught me again not to be first on the bandwagon with MS, hadn't done that for years) and bombed the preinstalled setup, and I had to wait 6 weeks for a new disk from MS. Back to NT...
Then the second try bombed because of some network problem setting up the clients which I still can't figure out what went wrong, but made me reinstall...with the new Disk 3...and it still wasn't working right, and I ran out of time...
Three times was the charm; the third time which was 6 months and three shows later all went well and things have been OK since then.
I couldn't believe that ancient NT PDC came back every time I replugged it into the network... I was prepared for major restores every time.
Of course the the PHD was my supervisor and was not pleased by all the delays, which was duly noted at my next performance review even though I had not lost a scrap of data during the long and painful process...
Next time I will insist on some lab testing first (I've been saving our older unused machines for it since then).
by vrinda
by jfk8680
It wasn't my mistake but we lost a complete days work:
We used a product called Standby Server for Netware 4.11 back in 1999. You could hook up two (different) servers and Standby Server made the disks in the second (standby) server available as local disks so you could create a mirror.
Everything was working fine but we were having some performance issues so our vendor decided to upgrade the NIC drivers. We were asked to break the mirror in the morning so they could send over an engineer who could upgrade the drivers at the end of the day. After he upgraded the drivers he recreated the mirror but did it the wrong way around. The data on our production server was overwritten by the 8 hours old data from standby server. Of course there was no backup because he simply had to update the drivers for the NIC's.
by bigwillie
by rycherulz
If you work in the industry long enough, you will absolutely have some world class screw ups.
1. Back in 1998 or 1999 (I've blocked the date from my memory), I was working on an 8 station Win 95 peer to peer network running Peachtree Accounting software, with the database being on the secretaries drive. They bought another Win 98 desktop to serve as a print server and file server, which was just basically a PC that no one used with shared folders and printers. I set it up, copied the database over to the new machine, and pointed all their Peachtree software to point there, and all was well with the world. One LITTLE problem; I forgot to move the backup software (a Syquest drive, remember those?) to point to the new database to back that up. SO, what ended up happening is that it backed up the old database that no one used over and over again. 3 months later, you guessed it, their new database developed a corruption in it that was unfixable. I showed up, went to restore the backup to the server, and wondered why the date on the file was 3 months old. They spent the next few weeks pulling up the paper invoices, and re-entering all their data. Woopsy!!
2. Later that year, at the SAME COMPANY (they loved me after this one), there was a lightning strike on the building. It fried most of their network cards, the hub and a few video cards. While I was replacing the hub, they decided now would be a good time to upgrade it from a 10BaseT to 100BaseT. No problemo, the nics were all 100BaseT compatible, so just plop in the new hub and voila! Ready to go!! Ummm... no. When I had first put together the network, I had used some of the old network wire from their Novell 2.x network. I had put Cat 5 wiring to add new machines that weren't there before to the network. However, the old wiring was Cat 2, which can only handle 10baseT connection. When I tried hooking up the 100baseT hub, the machines were getting/losing connectivity faster than I could keep track. It took me a full day to figure out that I shouldn't have used the old wiring. Needless to say, they never called me after that.
My knowledge and troubleshooting skills have greatly improved since those many years ago.
by mikewbc
I work at a Christian Camp and Conference Center that has a network of 4 servers and over 50 clients. We are a pretty conservative bunch of people and were excited about the new features of our new antivirus software.
When installing and configuring Trend Micro's Client Server Messaging antivirus suite, I waited to activate the new e-mail filtering until after I made sure the desktop antivirus and Exchange antivirus systems were working.
One day the executive director got a pretty graphic pornographic spam e-mail, and he told me that I needed to get the filters configured and working "TODAY". I quickly glanced at the manual and configured it that day. Everything was fine, but a not perfect. I went in to tweek it a little and saw a check box (with a check in it) that said "match must be exact". I thought that if I unchecked that box it might get a little more selective. I went ahead and unchecked it and went home. That night I e-mailed a file to my work address and got a strange notice back that the e-mail I had sent contained language that violated the organizations e-mail policy and that it had been deleted.
Early the next morning, I got a call that our e-mail was down and that we were getting all sorts of calls from Pastors, Bishops, Board members and a lot of Donors that had the same message I had gotten the night before. They couldn't figure out what "offensive" language they might have used in an e-mail. Some hadn't contacted us by e-mail in months!
Come to find out, not only did the filter stop incoming e-mail, it also went through the whole exchange store and deleted any message that it thought was offensive. It sent a copy of the message to the sender, recipient and administrator accounts. By unchecking the box, the filter was now seeing words like "Ass"istant, "Ass"ociate and hundreds of other words as offensive. That was bad enough, but the notification had a word in it that the filter thought was offensive so it deleted the notice, sent a message to the sender, recipient, and administrator - a never ending loop that literally sent out hundreds of thousands of e-mails before it crashed the server. It also deleted attachments and e-mail that people had saved in their exchange account.
It took me almost a week to restore "most" but not all of the old e-mails and attachments and to delete thousands of notices from the administrator account - all because of 1 little check box.
One good thing did come out of it, no one ever rushes me to implement, install, or modify a new program until I have a chance to learn about it and do a test install!
by alxnsc9
Dear colleague,
Don't you think you have fallen into an idiotic program design trap? It is not your fault at all.
A lot of antispam software is the same way faulty. Most of the so called developers have not any linguistic literacy but write linguistic filters. They can code but not design... The box you mention is an example of stupid design.
by oz_media
Except I formatted the backup itself. DOH!
It was one of those, look around and if nobody's lookig just restore a differet backup.
"Your file is missing? Are you SURE it was saved properly? Oh, thats too bad, I'll see if I can find it, but no promises.....oh that's okay, you're so very welcome, it's my job afterall. No don't worry about it, okay I'll let you buy me ONE drink but I am not sure if I can restore it or not. Just for my troubles? Oh, how kind of you! Do you have any girlfriends that want to join us?"
Then there's the time I sent out copies of a George Carlin standup special (VERY filthy!) to clients waiting for promo material, instead of our very own Julian!
by rizwandean
by swlchris
I forgot all about them just the other day . Customer had guts of a Compaq torn out and reinstalled to a new Intel box some time ago. He managed to crash and thrash the W2K that had been installed.
I get it , no problem reinstalling W2K, huge problem trying to figure out what driver to get.Spent more than a few minutes trying every single freakin Compaq network card driver.
Fellow tech arrives on scene, pops in a modem driver. voila, new device found, nore more pci device? in device manager.
Man I felt so stupid , I walked outside for a few minutes , lit a marlboro and thought about becoming an MTA employee.
Customer was waiting too, good thing he wasn't all pissy about it....but then again I'm not the one trying to use AOL dialup either.
Bottom line is, sometimes those freaky little cards got secrets....I never wrote down half my stuff on em. Any bets on how many notebooks I bought that night at the dollar store?
by sql_joe
When I was still fairly new to being a DBA I once changed the quantity on hand of all parts in our parts master table to "3". That was almost a million records. 3 cabs, 3 nuts, 3 washers, 3 engines, ... you get the idea.....On DB2 for MVS it took mere seconds....and hours to recover!
George
Sort by Trending