General discussion

  • Creator
    Topic
  • #2290428

    What’s your biggest IT blunder

    Locked

    by newsletter ed ·

    Some TechRepublic e-newsletter subscribers may have received an extra issue by mistake this week. During testing of our upcoming database upgrade, one of the staging components hit our live production database and resent messages from last week. We’re sorry for the extra send, but as IT pros you know stuff happens. Help us feel better by telling us about your top IT blunder.

All Comments

  • Author
    Replies
    • #2710225

      Reply To: What’s your biggest IT blunder

      by halinator9000 ·

      In reply to What’s your biggest IT blunder

      My biggest blunder would be using a computer in the first place … it was the reason for all subsequent blunders.

      =)

      • #2710149

        The wonderful command call “rm *.*”

        by maizan.dasimin ·

        In reply to Reply To: What’s your biggest IT blunder

        My group had this totally difficult customer project that took us 2 entire days and nite to complete. I took the night shift the following day from my colleague and took over the job from him.

        Still dazed while working at nite shift…I login to my unix terminal, proceed to the project directory and apply “rm -rf *.*”…

        Cold sweat runs through my spine after I realised of what I’ve done… I scrambled to re-do as much as possible, and work like an Ox for the next 12hrs… what a night !

    • #2710078

      I thought I saved the CAD system…..

      by jemslie ·

      In reply to What’s your biggest IT blunder

      In 1999, I was preparing to test the E911 computer aided dispatching system for Y2K by advancing the system date into the year 2000 and making sure our system would function properly. Of course I made a tape backup of the entire application system first, then advanced the date, tested and determined that everything was working properly. Wondeful! Then I went to restore the system back to its original state only to discover, much to my dismay, that the files were not there! An IT person’s worst nightmare come true. Horrors! I knew I had backed up the files, I just knew it! How could I lose these files!? Ok – time to panic! Then I looked at the contents of the tape again and realised that I had not initialized the tape and that my backup files were safe and sound, but were further back on the tape. Ok – time to jump start the heart again. Whew! It was the longest hour of my entire career.

    • #2710073

      Disaster Planning

      by tomasomaguire ·

      In reply to What’s your biggest IT blunder

      Three crows of the cock. My most memorable IT scenario is of a post IT type server farm of NT 3.51/4.0 With Xchge 5.0 across 3 servers with internal and external e-mail servers. This was pasted together by gamers and code heads so that you could never turn any one server off without downing the whole e-mail system. This was extended on a Portmaster 2e Frame Switch. If the system got stuck or the Frame switch went down I would reboot or throttle ( yes you can!) or open the Frame switch and reseat the memory. The switch went down 2x each time longer than the previous. The server farm was spaghetti and I had made several drawings to figure how they had slung the Xchge server across the 3 servers. Also there were scsi shoe box raid drives which contained legal docs, and c code from an IP dispute ongoing et a DAT backup tape system. I spent many nights trying to unravel this mystery.
      2 is the number. At 2 you stop and give the urgency emergency. I kept trying to back up and re-animate the thing and once we had to move the whole system including DT’s and Workstations. The Blue Screen of death appeared over and over and the tape system was not relaible. The true response is to stop and cash out! Make immediate contingency plans. Put a price on everything. Cost it out and that morning you go to IT and tell them that you can save the data and that is all. You need a new set of tapes and a DAT drive.
      I have a 90% failure rate in this business. What company worth spit won’t pay for the 10% success I got?

    • #2710062

      My big oops!

      by rigmarol ·

      In reply to What’s your biggest IT blunder

      I was working on how to upgrade the entire office to MS OfficeXP from Office 2000. I had done all the reading, made all the group policy adjustments and all I had to do was apply the policy right? So I create a test group, put them in a test container, put my global group policy at the top of the tree structure and rebooted my test PC with it’s test user. Hey, worked great!! perfect I’m ready to go live. But decided to wait until tomorrow so I could make an announcement what was going to happen when.

      I came in the next morning to a flurry of help desk calls. It seems when everyone turned on their PCs that morning MS OfficeXP was auto installing on EVERY PC!!!!! Oh my Gawd, what happened?

      If you read carefully you’ll see above, I applied my group policy changes to the TOP of the tree and not to my Test container.

      Good news; it all went without a hitch, everyone was fully upgraded withing 2 hours and the only negative was a terse email from the boss asking why he didn’t get the email announcement…

      Whew!

      • #2710484

        Whoops

        by supportoranges ·

        In reply to My big oops!

        Tried to gently pry off an AMD K5 chip and cracked it in half. Oops!

        • #3295450

          Wasn’t mine but….

          by jamesrl ·

          In reply to Whoops

          I was working at a major corp as desktop tech. Department was getting Macs, replacing PCs. We installed the Macs and left the PCs there for a week while they transferred data to a common drive, or backed it up onto floppy.

          One user thought he was computer smart, his brother was certainly a techie. He asked about printing – he had a laserjet, and didn’t like the idea of using s shared network printer a few feet away. I told him his laserjet just wouldn’t work with the Mac(back then just parrallel).

          He noticed that the parrallel cable fit nicely into one of the ports on the back of the Mac. It was the SCSI port -built into the motherboard. On powering up, he blew the motherboard of the Mac.

          Needless to say, as someone in the department who paid for new computers, and knowing that I couldn’t fool the onsite HW repair person from the vendor, I had to give the guy an earful. Because he refused to believe me he fried a very expensive motherboard.

          James

    • #2710001

      Remember to test backups!

      by midweststeve ·

      In reply to What’s your biggest IT blunder

      I had just taken a role as a LAN administrator, and had completed installation of a new LAN with about 20 users. I had setup our backup software to backup all files using the “*” command. I thought this would be fine, since whenever I had retrieved all files from previous installations I used “*” instead of “*.*”.

      The server’s hard drive crashed, and I went to the backup tapes, only to find that I had backed up only those files without an extension for the past two weeks. All that work had to be recreated, and I felt like such a fool. The only thing that made me feel just a tiny bit better was the backup software vendor (a major company that I’ll let escape injury in this post) acknowledged this as a bug and fixed it in a subsequent release.

      This has served as a valuable lesson now that I’m in management, and can help my team know the perils of not fully testing. And they don’t let me near the backup software any more 🙂

    • #2709570

      biggest IT blunder

      by abme75 ·

      In reply to What’s your biggest IT blunder

      My biggest IT blunder is getting hacked. Sad..sad..day..month… Hey, Can you guy’s tell your partners to take it easy on the email ads.. Good God, How did they get my full legal name.. 🙁

      • #2709545

        What’s that command again?

        by portypicker ·

        In reply to biggest IT blunder

        This is one for the Pick/Multi-value people out there… there must be some?
        As a youngster to the trade I was in my first Pick job and learning my way around a system when I decided to delete a ‘Proc’ item (that’s a ‘procedural’ item, JPL sort of thing). So I used the command ‘DELETE-FILE PROC item’ (you know, delete the item in the file PROC called ‘item’?) – oops! That should’ve been ‘DELETE PROC item’… I’d deleted the entire file. Suddenly the phone is ringing off the hook – no-one can access anything (the entire menu system hangs of the PROC file), processes are falling over, basically the computer world had ended!
        Fortunately, I had a full system backup and could restore the file – only took an hour or two – and I only had to redo all the changes I’d made that day. Haven’t done it again, strangely enough!

    • #2709537

      DOS / Win 3.11

      by kazemoth ·

      In reply to What’s your biggest IT blunder

      Way back when …
      Doing some housekeeping in DOS and needed to copy some files to floppy. Found floppy and checked if it was empty – no. No problem – del *.*
      You can guess what happened. I was in the Windows dir at the time and had to reload off floppies, etc. No great blunder, but was the start of always checking what dir I was in before perfoming cmd line instructions.

    • #2709424

      Double check who your sending that email to!

      by chris ·

      In reply to What’s your biggest IT blunder

      I am IT Mananger at a offshore shipping company.
      Over the past 3 years I was tasked with being the first point of contact with a third party software development company which was developing a ?100,000 database for our company.
      Nearing the end of the contract I had serious issues with the dev company in the way it was handling the testing etc of the database. I sent a rather informal email to this affect, explaining what a ‘t*****’ one of the programmers was to one of my colleages in our London office, however with Outlooks ‘great!!’ autocomplete feature I proceeded to send the message direct to the programmer instead (who had a very similar name to my london colleage).

      Needless to say it created a very awkward situtation with the dev company to the extent that the relationship broke down and they no longer want to work for us! Hmm. I got a few stern words from the director but in the end he agreed he was a ‘t*****’ anyway!!

      Moral of this story…Check and check again the address who your sending emails to before hitting that send button!

      • #2709358

        Similar story

        by jamesrl ·

        In reply to Double check who your sending that email to!

        Once had a student come to me, breathless, wide eyed, asking if we could delete an email off the server that he had just sent. He had sent a somewhat um…riske…note to a fellow student or so he thought. When he checked again he noticed the autocomplete had pulled up a different name.

        We went to the server and discovered it had already gone to the remote mail server, and as it was after 5 we could not get ahold of the remote admin, unless we paged, and that would have raised more questions.

        The receiver did not enjoy the note….student had a meeting with HR.

        James

    • #2709351

      Managing a rogue server

      by jamesrl ·

      In reply to What’s your biggest IT blunder

      At a large corporation, had a project with very little budget to create a project related document depository that conformed to our project management process.

      We didn’t put the server in the datacentre, we had a spare office we used as a lab to house the productions and development servers. But before we bought our shiny new server with RAID, we were using a workstation(Compaq Deskpro with SCSI) as a server. We had a tape backup that I used religiously. But one of the project leads was a little paranoid about tape, and asked if I would take his 1GB Jazz drive and back up the data files.

      I agreed to do it after hours. That Deskpro had an onboard SCSI connector, so I plugged it right in. The Jazz drive had an automatic sensor to determine whether it needed to terminate the SCSI chain. What I didn’t know is that it didn’t play nice with the onboard SCSI, and the boot drive got a bit of a hit. This was before Ghost and other fun utilities, and the backup covered the data only. I had to scramble around rifling through collegues desks to find the right version of Win NT(with the right service pack) to do a repair after getting the infamous blue screen of death. I found this out through trial and error having watched a number of attempts fail.

      After a longish night, the system was up by 9 the next morning. And I invested in a real server, with RAID. Never trusted a Jazz drive again.

      James

    • #2709249

      Hangs head in shame!!

      by guruofdos ·

      In reply to What’s your biggest IT blunder

      Getting a call to say ‘hard drive full…need upgrade’. Turned up with a brand spanking new Maxtor 20Gb drive and a copy of Maxblast. Hooked both drives onto the computer, then did the automated ‘copy drive’ thing…only I copied the destination onto the source instead of vice versa and completely erased 10Gb of customers (mission critical) data. Fortunately, they had done a backup onto Zip100’s a day or so before, and all they lost was a couple of Word documents. Luckily they had hard copy, so I scanned it in using OCR and regenerated the missing files, on my own time. So much for telling the wife I’d only be out an hour…which was the BIGGEST blunder!!

      • #2717842

        Telecom cabling headaches

        by oz_media ·

        In reply to Hangs head in shame!!

        Years ago, I was onsite during a new phone cutover. It was a wierd location for a movie set (Lake Placid with Bridget Fonda, that’s another story altogether though) after the cut over was complete, everything seemed fine.

        We watched a few hours of shootig, pigged out on Craft Services and headed home, easy day, dinner with a Fonda and some autographied pics, COOL JOB!!

        The next day, I get a call saying that none of the calls TO the director were coming in, they could dial the number and it would ring away but no awswer. Een though they could see him across the room, his phone didn’t ring.

        So we went back in and ran cable tests, on the bix racks and all toned out fine, meaning the cabling was OKAY.

        When toning the jacks, nothing at the duirectors desk.

        So we ran NEW cabling through and STILL no go.

        So during lunch (again pigging out on Craft Services, Oliver Platt is hilarious!) I noticed the PAYPHONE that had been placed on the set wouldn’t stop ringing. Seeing as our staff had cabled it, I went and picked it up. It was someone looking for the director?

        So I game the prson the directors local and direct number, the payphone rang immediately. At that time, the little lightbulb in my head started to glimmer ad flicker .

        We then moved the punchdown from the payphoe to the correct extension and all calls came in to the director as expected.

        So we had been so chummy with Bridget Fonda and Oliver PLatt, that they asked us to spend the rest of the week on set and showing them around town between shoots.

        MY PLEASURE!

        • #2717821

          Cool!

          by tomsal ·

          In reply to Telecom cabling headaches

          I agree cool job assignment Oz. I think it would be neat to do some IT work for something like that. I just deal with regular corporate types instead.zzzzzzzzzzzzzzzzzzzzzz.

          Though I will say…I must confess though I thought Lake Placid was a awful movie.

        • #2716550

          Oh and it was.

          by oz_media ·

          In reply to Cool!

          Lake placid was such a horrendous flop! What a crock of garbage.

          It was cool to meet the set designer who and photographed a local lake, then built the identical set in th Surrey flats (an Auto wrecker/parts neighourhood). Trees, lake Croc etc. It was really neat, to watch actually. Bridget Fonda was a total sweeetheart who doesn’t act like a rich snot at all, Oliver Platt was just plain funny.

          I have also been on a few other shoots around town as they usually rent systems from a local company I used to work for.

          My friend used to be a double for Dean Hagland on the X-Files so I got to meet all those guys a few times too. Now it seems they shoot so many movies around the Mainland and Island that you hardly even notice anymore.

      • #2717822

        Communication

        by shook4brains ·

        In reply to Hangs head in shame!!

        As an IT consultant, I am always working odd hours. My biggest blunder of all time is telling the family, this won’t take long, I’ll be back in a couple of hours. Then I get the the job and uncover a much larger problem. Hey honey, I’ll see you and the kids next week. Bottom line, KEEP THE WIFE INFORMED!!!

        • #2716607
          Avatar photo

          I can relate to that one

          by hal 9000 ·

          In reply to Communication

          Only last week I got a call out and said I’d be back in a couple of hours. Well as usual for anything IT related it took quite a bit longer than expected as well as another job coming up but each time I did ring the wife to say I’d be a bit latter.

          However when I eventually returned home some 14 hours latter I was greeted with a stony silence and just one question “Can I have my car keys back!” seems that she had placed her keys into my cars glove box for safe keeping the previous day and while she had spare keys for her car there where no house keys as I was supposed to get a few spares cut that day.

          Col

        • #2716549

          I am usually the opposite

          by oz_media ·

          In reply to I can relate to that one

          As a mechanic, I have learned estimation and diagnostic skills well. I usually over estimate time and it leaves me room to dawdle.

          If I say I’ am goig to be on a six hour or two day job, it may take 2-6 hours ad the rest is bar and track time.

        • #2717444
          Avatar photo

          Well Oz you’d love one of my more recent new clients

          by hal 9000 ·

          In reply to I am usually the opposite

          I was told that all I needed to do was install a DVD drive in a Compaq Evo. 30 minutes to get there 15 minutes to install and make sure everything worked and 30 minutes home. That was the idea anyway when I got there I found no software on any of the workstations the “new” network cables where hanging out of the wall on the cables the network was not set up at all other than allowing e-mail and even then all that was installed was a ADSL modem feeding a hub. And best still they expected to have to e-mail files from one workstation to another 10 feet away, {Talk about a “Professional Installation!”}

          Anyway 4 weeks latter when I left that place they had everything working and all the software that they really needed OH I forgot to mention the fact that they had superficially ordered a second monitor for the salesman to show any customers on the other side of the desk what was available which of course was not possible to fit to the Compaq, but they where sold the hardware none the less.

          I particularly liked the person who was hired to run the cables as that is all he did and then walked out the door with a short message “I’ll be back” well it was 4 weeks latter and no message or anything but he did walk in the door just as I was leaving and complained that he could have provided all of the stuff that I had and was quite out put because he hadn’t got the added work.

          I suppose some companies can afford to run for weeks without any real software and run up horrendous e-mail bills just forwarding files between workstations. He obviously thought this way.

          Col

    • #2710473

      Where to start?

      by saintgeorge ·

      In reply to What’s your biggest IT blunder

      I might have made a career out of blunders, if anybody paid for them.

      Working at college for the Systems Deparment, I had been at the job for two years and I was less junior than the new juniors. I was not senior, yet. Kind of a medium. The juniors used to keep the air conditioning off though us mediums and seniors kept telling them to turn it on. One day I arrived to find the heat was stiffling. I stomped over to the electricity board and turned on the AC switch. Nothing happened. Obviously, it was broken. Lucky me, I hadn’t dress them down, how embarrassing it would have been! I glanced over to the other side of the glass partition, and I saw a bunch of the juniors looking all over the TI990, pressing buttons. Oh sh.t. I hadn’t turned on the AC but turned off the general switch. I had switched off the TI990 and the seven students terminals it served and the IBM320 with its five administration terminals. Survival took over and in a split second I had switched it back on and stepped out of the booth asking what the heck had just happened. Nobody was sure and we concluded a glitch in the electric circuitry had been responsible. Some glitch.

      It was the first but there were more to come…

      • #2721476

        Maybe it taught them a lesson?

        by hotshot3000 ·

        In reply to Where to start?

        >One day I arrived to find the heat was stiffling. I stomped over to the electricity board and turned on the AC switch. Nothing happened. Obviously, it was broken. Lucky me, I hadn’t dress them down, how embarrassing it would have been!< Sounds like someone fell down on labelling switches! I sure would not feel guilty about that. Did these folks not also know that heat is bad for computers?

    • #2710313

      Whoops

      by tech_guy1 ·

      In reply to What’s your biggest IT blunder

      I was in the middle of converting our ERP system over to multifacility during an implementation. I started finishing up around 2 a.m. and needed to do a mass change and set the users into their appropriate facility. When doing the mass change, I hit the wrong records and hosed all the login accounts including the Admin accounts. Had to recreate all the logins, settings, security, printers for all accounts. About 3 days worth of work.

    • #2710276

      Is it Hotswapable ??? (let’s try)

      by julien.grelet ·

      In reply to What’s your biggest IT blunder

      I was hired for replacing a guy in an insurrance company. He was so bad at work, that i try to convince him that the disk in the raid array of our Exchange server where hotswapable (it was for joking). Bad idea, one day, a disk felt and he try to replace it. Damn, i’ve neverseen an exchange server doing fireworks !!!. I spend my week end to rebuild the Ex server and explain my boss that the server room was full of electricity in the air

    • #2711996

      Cut the cables — NO not THAT ONE!!!!

      by tomsal ·

      In reply to What’s your biggest IT blunder

      In 10 years I’ve made a couple blunders, but the latest and greatest (or should I say “worse”) happened just last year.

      Our entire facility underwent massive rennovation, on a scale you could honestly call it a re-construction instead.

      Well the many teams of contractors we had working here made for a very busy atmosphere for about 8 months straight. When it came time for lines — either telephone, data, cable or in a couple places even electrical to be cut — everyone from the CEO to the foreman of the contractors doing the job turned to me to ask what cables are good to go, which ones stay and which ones need to be re-routed.

      Being the quintessential highly detail oriented geek that we in IT are, I of course had all kinds of documentation and nifty layouts of the building pre-deconstruction.

      Well after working a 16 hour shift on a Friday, some contractors were working late to make their schedule for the following Monday.

      Needless to say the end result — I was tired, I was hungry, I wanted to go home — I told them to cut a group of data cables that I “Thought” should of been dead.

      I thought wrong. The data connections for two whole departments went dark. One these departments being a heavy revenue earning one I might add.

      To say my face glowed red with shame, embarrassment and anger along with my stomach twisting in knots would be the understatement of the year.

      The mistake was made around 4:00 PM, by 11:30 PM that night I had everything fixed, it looked sloppy as hell mind you but it worked.

      When I got home I just wanted to crawl in bed for a week or two. lol.

      • #2723044

        Familiar tune!

        by guruofdos ·

        In reply to Cut the cables — NO not THAT ONE!!!!

        As well as computer work, I get to do a lot of AV installation and cabling jobs too.

        We had a big job on recently on an RAF base near us. Their old Barco 3 CRT projector was getting a bit tired and they wanted a complete refurb of a training and lecture room. The new LCD projector had to be mounted in a different location from the old projector to fill the existing screen, and we duly lowered the old projector 25 feet to the ground. The power for the projector was fed down the ceiling mount tube and this had to be removed and the ceiling made good prior to hanging the new bracket and projector. I thought I’d traced the power cable down the wall trunking to the lecturn and when I spotted a label on the plug saying ‘Projector Power’ I thought I’d hit the jackpot. I removed said plug from the wall, then climbed back up the ladder to remove the old tube and bracket. Rather than fiddle the cable with the IEC connector up the tube I decided to simply cut off the plug as the whole installation was being recabled anyways.

        Bang, flash…puff of smoke and sudden darkness. Two big chunks literally melted out of the jaws of my cutters and another even bigger crash as I fell the 20 feet or so from the top of the ladder. A minute later, a big burly Squadron Leader poked his head round the door and asked if I had any idea why the coffee machine and the TV in the Rec Room had suddenly ceased to function.

        Turns out, the plug marked up as ‘Projector Power’ belonged to a 35mm slide projector on the rear wall of the room, and the main feed to the Barco video projector had been spliced directly (naughty, naughty!) onto a main 60A feed on the main distribution board for the building!!!

        Moral of the story…never trust the previous installer!

    • #2711990

      Don’t start on Friday afternoon

      by grant@rb ·

      In reply to What’s your biggest IT blunder

      (This is the only blunder I’m gonna put in writing. You think I’m crazy?)

      I got a brand new machine for one of my users, a complex, tempermental, and high maintenance female. To get her out of my hair, I told her she could go home early b/c I was going to swap out her computer. She gave me her login and I told her we’d change passwords next week.

      So about 3pm on Friday, I swap out her machine. I had all the apps ready to roll, heck, I even had all the desktop icons in the same places! Good tech that I am, I tested her printers and checked her mapped drives, nada problema.

      I’m getting ready to go, got a hot date, but then I remembered one of those desktop icons connected to a database and I hadn’t tested it yet. No worky. I fire up the original machine, that icon works. I’m sweating, I gotta go, if I leave my user w/ a bad icon, I’m gonna get roasted.

      Bag it, sez me. I put the old machine back and leave a note, saying we’ll talk next week and reschedule.

      Moral of the story: don’t start jobs on Friday
      -unless you want to work all weekend.

    • #2711979

      I thought you knew the password

      by service junkie ·

      In reply to What’s your biggest IT blunder

      One evening I was working with a colleague on an NT to Win2K domain upgrade project we’d planned together. We both worked for the same firm and shared a mutual respect as well as mutual habit of covering all the little details. However, we had not actually worked together prior to this project. Therefore, we were both beaming with enthusiasm over the opportunity to execute this project together. This first night called for removing a server from the domain and performing so tasks which I don’t recall at this time (this was about 2 yrs ago.) But I do recall all the data was still on the drives.

      So, we’re chatting away… everything is proceeding very nicely. After removing the server from the domain, my colleague attempts to login and turns to me for the local admin password. I tell him the password is “blahblahblah, just like all the other servers”. He types in the password and gets the NT invalid password message. We laugh as he joked “wouldn’t it be something if we didn’t have the password” – hahaha, both of us convinced that he’d just mis-typed it the first time. The second time the NT invalid password message popped up, we weren’t laughing any more. We both had that “Oh $h*% look. We go to a different server, and logon locally using the same password. Damn, someone had set this one server to a different password.

      After two hours of phone calls, couldn’t find anyone to fess-up changing the admin password. But, we did manage to snag a copy of Bluecon from another associate a few miles down the road. Thank God it worked. We were able to reset the password and proceed with our project task.

      Now, I login locally every time before removing a server just to make sure I know the “current” local admin password.

    • #2717851

      TRAINING DISKS

      by oz_media ·

      In reply to What’s your biggest IT blunder

      I was given the task of building a set of training disks for a client.

      I created full AutoRun studio’s with FLASH itro’s, AVI product info etc.

      When I sent them to the 10 sales reps, I got a call back asking where the product info and demo was. I checked MY ddisks and found I had burned TWO George Carlin standup comedy specials (VERY adult humour!), a Rowan Atkinson standup special and another comedian (name slips memory now).

      So instead of sending out this fabulous creation I was quite proud of (pro-production by local video crew, music the whole whip)I had sent out VERY coarse Adlut standup comedy disks.

      They all had a laugh thankfully, and nobody was offended. I got the RIGHT disks out and they were very pleased, nobody ofered to give me back the other disks though.

      So I guess what I learned is that if you are gonna screw up, at least make it funny.

    • #2716622

      Self Spamming

      by paul.kiser ·

      In reply to What’s your biggest IT blunder

      I was lead for a system administration team working on Sun systems connected to our intranet which was running MS Exchange. Our group had just had a name change and we had requested our group email address be changed. Help desk replied no problem. They were going to keep both group names active a week to make sure everything worked as planned. On the Friday before our scheduled Saturday maintenance, they called back and said they couldn’t complete the process until Monday. Okay, shouldn’t affect anything, right? Wrong! Our Sun server was set up to email all “root” messages to our Exchange group account. Also, our maintenance scripts would automatically reboot the server after running the backups and upon reboot, the startup scripts on the would then send an email to root giving the startup status. Since all root mail goes to our Exchange group, it was sent out. Exchange couldn’t find the group (we had changed our group account on the Sun server earlier that week). Now the fun starts. It returns the message to “root”. The Sun receives the message destined for root and then immediately forwards the message to our group account, which doesn’t exist on Exchange. Exchange then returns the message. When our mail queue ran out of space on the Sun, we started having some real fun. Oh, did I mention that our two rookie administrators were doing maintenance for the first time by themselves? After several phone calls to me trying to explain what happened, I got them to shut down the Sun to single-user and delete all messages in the queue and then change the alias back to the old one so new messages would go to the right account. When I came in on Monday, I had over 18,000 messages awaiting me. Fortunately our manager was pretty understanding only because the server was scheduled to be down during that time. It was definitely a good lesson to learn and exposed my two rookies to some hot-seat troubleshooting.

    • #2716598
      Avatar photo

      Well I can honestly say my biggest screw up

      by hal 9000 ·

      In reply to What’s your biggest IT blunder

      Was not catching that apprentice and killing him when I tried.

      The first time I actually worked with this kid was in a building 20 stories high and with several thousand workstations. What should have been a simple .5 day job of installing a new workstation on a Unix platform turned into a complete disaster. Apparently when the new workstation was connected everything went down in a screaming heap and that was only the computers what came from the users was a different story all together.

      Anyway in an attempt to get things fixed quickly I disconnected the new installation replaced the obviously damaged bits that I could see and fired the beast up again here we are talking about one 9300 IBM mainframe with about 20 frames from memory and eventually everything worked again. But then the fun started as the apprentice came along saw his work disconnected and replaced the plugs. Down went everything again and this time the users where not the slightest bit happy I’ll leave it to your imagination about some of the complaints that started coming through but this time I chucked the apprentice out and redid all of my previous work only took about 3 hours and God only knows how many $ in parts. But that paled in comparison to what was being lost in downtime.

      Eventually when I got around to checking the apprentices work I found that he had run a lead from the sub sub basement to the 18 th floor and placed one of the plugs on 180 Degrees out so what should have been a signal line was in fact a power line no wonder about the damage that had been done.

      Eventually the apprentice returned and I casually asked just how he had checked the lead before fitting the plug his response was with a signal generator so I asked just how he had checked this lead apparently he connected all the lines together and sent a signal up the line and when he found the line at the other end putting out the signal that he was sending up the lead he just placed the plug on as his supervisor had instructed {incidental that wasn’t me as I was only called in after the first shut down.}

      I then suggested that he only send a signal up one line and work that way as it didn’t require all the lines to have a signal present to identify the required lead. His response was well lets say less than helpful and he even suggested that while I was an old dinosaur who knew nothing it was actually him who had saved me all the work that I’d just spent the best part of the day performing in the most condescending manner. At that point I just picked up a 18 inch number 1 Phillips screwdriver and proceeded to chase him out of the building with a solid intent of driving the screwdriver through his chest and using it to nail him to a wall hopefully with his feet several inches off the ground.

      UNFORTUNATELY I DIDN’T CATCH HIM and he gave me lots of other nice little repair jobs further down the track particularly when he became a fitter as he would go in with a crew and install the hardware and then complain when it didn’t work. It was at that stage that I got called in and I’d find a nice new installation with no functioning parts and best of all no idea of just where cables ran as this wasn’t an important thing to him or the people he worked with. His idea was to install the things and then it was a “Techs” job to get them working. I can honestly say it would have been far easier installing these things myself as it would not have taken as long and I’d have had a nice wiring digram after wards as well.

      Now if only I had caught him early on in the peace.

      Col

    • #2723979

      “not =” is not “=”

      by marco schumacher (at biznesslegion) ·

      In reply to What’s your biggest IT blunder

      It feels better to talk about somebody else’s problem, so I will tell what a fellow programmer of mine did (a long time ago).

      He was supposed to clean out obsolete accounts from a bank’s database. Unfortunately, instead of deleting all records where “status not = active”, he … you guessed it … deleted all those with “status = active.”

      This was in the good, ole’ days of batch, so this happened overnight. Fortunately, a colleague noticed the problem first thing in the morning and ordered a database restore.

    • #3296577

      Read the Project Plan you wrote

      by steven131313 ·

      In reply to What’s your biggest IT blunder

      I was in South Australia developing and implementing a new system for the utility company. We used test servers as part of the implemetation was to install two new ones. Software tested just fine, specs were good, still in scope so we set our implementation date. The day came and all was well until we realizied that we had not ordered the two servers that were definitely in the project plan and on all of our task lists.

    • #3296572

      The Initializer

      by virtualgardener ·

      In reply to What’s your biggest IT blunder

      I was troubleshooting some scsi errors on our backup server. We have a FiberChannel Array of drives attached to it with some non-critical data stored there. I disabled the controller and powered off the array to see if the errors would go away after the reboot. They did not. Obviously the errors are coming from something else. Eventually I traced the errors to another controller, and then I turned the array back on. I went and told my boss that I had figured it out and he told me I needed to initialize the software that controls the array. Just turning it back on doesn’t do the trick. Sure enough, when I went back to the server, the array was not available. I opened the FiberChannel Assistant software, but I couldn’t see the drives even from there. I told the software to search for available drives. Well, the drive arrays are actually available through IP. Unknown to me, the software went out and found the production array attached to another server. I didn’t notice that the IP address was different from the machine I was on. The more observant of you have probably figured out already that I forgot to re-enable the controller card for the array attached to the backup server. So when I searched for drives, the software went out through IP and found the other production array. Being new to the SCSI world, and not knowing what I was doing. I then searched at great length for the initialize command. Found it and executed it on the production drive. It kept asking me if I was sure I wanted to do this. Of course I do. Didn’t seem to be working, so I told the boss I was having trouble figuring it out. He said no problem, I’ll take care of it. It’s 5:00 go ahead and head home. Well, he walked into the server room, and for those of you who may not know, initialize in SCSI speak means FORMAT. He just wanted me to open the software and it should do an autodiscovery, find the local array and make it available to the operating system. So a series of three mistakes caused me to reformat the entire production array. I went home, all proud and happy that I had finally figured out the source of the SCSI error on my backup server, and my boss, the Application manager and the CIO spent the next 4 hours on a friday night trying to get the production data back on line. My boss told me that everyone is entitled to one good screw-up, and this was mine. My new official job title is “The Initializer”.

      • #3295512

        Very generous

        by toucan ·

        In reply to The Initializer

        Many bosses would have given you the boot instead of a nickname

    • #3295513

      A bonehead manuver

      by toucan ·

      In reply to What’s your biggest IT blunder

      Aside from reformatting my own workstation while talking to the nice lady …

      We had an application server fry an array controller, the cold spare was on the bench so a quick rack switch seemed in order. Quickly unscrewing the server, I pulled the server (naturally in the top of the rack) out and discovered as it slide out and fell to the floor that the rail stops didn’t work. Missed my feet but I’ve nevver seen a server chassis bent like a potato chip before.

Viewing 22 reply threads