Community archive

Ask something
Discussion Locked

My biggest foul-up.

0 helpful
31 replies

by jardinier

Closed to new repliesThis discussion is closed. It stays readable, but it takes no new replies.

[Please substitute balls-up, cock-up, screw-up, faux pas, etc according to your preferred idiom]

I am going to cheat slightly by using a third-party example, because I have never personally managed to create a disaster on this scale, although hadfate decreed, I might well have been the actual perpetrator.

Whilst working in the Engineering Products section of AWA, many years ago, I was "witness" to the following incident. A large radio transmitter had been assembled, thoroughly tested andready to be packaged and shipped to its destination. However a supervisor noticed a form of wires was sagging slightly, and instructed an apprentice to install an additional clamp. The apprentice drilled into the aluminium panel, installed the clamp, and the impressive-sized piece of equipment was subsequently packaged and shipped to its destination in New Zealand.

However, when it was switched on for the first time: Ka-boom! the apprentice had inadvertently drilled through a form of wires on the opposite side of the panel.

So let's here it gents: about the time you brought down a whole network by dropping your screwdriver into a workstation .....

31 replies

by scarecrow71

Let me give you an example. Whilst working at a previous company, one of my colleagues needed to add another hard drive to his pc. Realising that he didn't have enough power leads from the PSU, he proceeded to build a splitter.

After building the splitter, he passed it around the department asking each person to check his handywork. In turn each man examined the cable carefully and gave it the thumbs up.

He then proceeded to install the new drive. When he powered up, no spin up, no lights, no sign of life from the drive. Maybe the hard drive was faulty, so he tried another drive. Nothing again. So this time he grabbed a CDRom drive which he *knew* was working and tried that. Again, failure.

At this point he decided to revisit thecable splitter. Upon further examination, he noticed that he had wired it up incorrectly. He corrected the error, and subsecuently realised that 2 hard drives and one cdrom had been short circuited beyond repair.

Every man jack in the department had checked that cable (including me!) and everyone of us had missed the glaring error.

Sometimes cock-ups are meant to happen. If we don't make mistakes, we can't learn from them.

by jardinier

I guess I should thank you for at least placing a posting on this "discussion" which I was about to declare still-born.
However, obviously your posting is not the kind I was hoping to attract.
In effect all you have implied is that you are infallible and would never have made the same error as the supervisor referred to in my posting.

However the whole point of the "discussion" is about giving examples of the how the human error factor caused a mechanical foul-up.

by scarecrow71

I'm not sure why you think my post was inappropriate.

I am not trying to say that I am infallible, quite the opposite, not only was I at fault, but every man jack in the department.

The example I gave was an example of how the human error factor caused a mechanical foul-up, but I was also trying to make the point that no amount of testing will trap every human error.

To err is human after all, but the true test is wwther you make the same error twice!

by jardinier

I lovingly assembled a computer from bits and pieces, including a CD-ROM and 5 1/4 floppy, installed the OS, tested it and everything was fine.
Then (because I am a bloody perfectionist) I wondered: "Did I fully insert the power plug to the 3 1/2 in?"
So, while not having an uninterrupted view of the connection, I removed the plug and reinserted it.
A minor explosive noise and a cloud of smoke. On examination, I found that two of the delicate pins on the 3 1/2 were inclined towards each other, so that when I pushed the plug in the second time, I created a dead short which I think killed every drive in the unit.

My only consolation was that my mentor, Patrick Li, who had run a full-time professional computer business for a number of years, invariably confessed, when I had done something stupid, that he had at some time done the same.

by generalist

Way back when the IBM PC was starting to get a foothold in business I ended up as the defacto PC tech in addition to my normal IBM System 38 programming job. The company I was with had one IBM PC with a 20MB hard disk that they got from one of their outlets. (The outlet purchased it but couldn't figure out what to do with the thing.)

One of the inhouse departments started using it for word processing using WordStar. At one point they had a problem with something.

One step of my solution for their problem involved formatting a 5 1/4 diskette. I went to the command line, typed FORMAT and let it run.

MS-DOS, at the time, didn't warn you if you were trying to format the hard disk. So I ended up formatting the hard disk.

Since then I've ALWAYS specified the drive when dealing with command line entries.

by lordinfidel

Like this weekend....

I was doing uhhh let's say, "penetration testing" from my home network.

My port scans were coming back with some false positives, so I decided to run the scan from outside my firewall. That is I connected my machine directly to the net, totally bypassing my firewall and NAT.

Well I accidentally left it on while I went out for a couple of hours. When I came back, my recieve and transmit lights were going off something furious.

When I looked into the active connections on my machine, I quickly noticed that I was 0wn3d!

Needles to say, I brought down my home network. Got rid of the root kits. Well not really, they now are part of my tool kit. For testing purposes of course.

So remember kid's, the net is evil and bad.

by road-dog

Foul up was not only embarassing, but captured on video as well.

I was working in payload processing on the expendable vehicle program at Kennedy Space Center. The project was to validate and calibrate a large spin balancing table to be used to assure radial balance of spacecraft.

Part of the test regimen was to place a 5 feet tall test weight on the spin table and place smaller weights around it and at different heights to see if the table would find the "out of balance condition".

Needless to say, I dropped a 150 ounce test weight on the table's surface. Since it was nickel, it made a ding about 3 CM long and about 5 MM Deep. The ding took about 10 manhours to polish and we had to start calibration over.

The German Tech Rep cussed me out in two languages and I was known as "butterfingers" for about a year!

by kethry127

Fortunately my biggest screw up didn't take down the network, but it was costly. We were lifecycling workstations and servers and had filled up a palette about 4 units high and 4 deep. To understand this, note that I'm a gal, and at the time the only gal on the team. I admit I had a 'tude about I can do anything the guys could. I tried to push that cart out of the server room to the dock. I forgot the ramp out side the door and hit a load of incoming hardware. When all was said and done, I'd demolished 5 new servers. And really amused the managers in the glass walled conference that I skied by prior to my unfortunate stop.

by jkaras

Wow, not to sound mean or anything but I would of paid money to see that! Did you get air as well as the servers? The aftermath of the chuckles had to bother you for a time. It's not the screwup that bothers you so much as the office story that getstold over and over. Great story.

by kethry127

I thank god this was before web cams, or you probably would see it. All I could think was this was going to be bad and then I noticed the managers in the glass conference room. When I came back in after the crash, they held up their scorecards. It was a popular topic at lunches for a long time, my IT spread it our other IT groups around the country. Did a great job of breaking down the barriers though. Fortunately, a coworker, a network cable running across the floor, and an intern with a full cup of joe took my crown. Scary thing is how many times I've met people who worked for that company and even though it's been years they still remember it. I get "oh you're that girl! I heard about you"

by road-dog

Working at a previous employer, we had twin t-1 connections between the NOC and a remote site manned by about 400 employees in a neighboring town.

One circuit went down and the remote router was having trouble because of the loopbacks the circuitprovider was putting on while troubleshooting.

My buddy decided to shut down the interface on the remote router to stabilize the routing environment.

Config t
int s0
Shut
............

No confirmation ever came. His look of confusion turned to a look of horror when he realized that he had shut down the operational interface. We had no dial backup management interface at the time (it quickly became a priority)

He dashed out the door and set a land speed record to the remote site and no shut ed the interface. Fortunately it was lunchtime and only porn and internet gaming was interrupted.

It was a classic example of cutting off the tree limb you're standing on....

by jardinier

My sincerest apologies Scarecrow71. I put the message on the wrong thread. It was directed at the posting above yours: "Testing" from DavidAndGoliath.

Actually your posting was the first proper posting, so thank you very much. And the original posting, which received no replies for a long time, has now taken off.

by jardinier

Some time ago a friend sent me the following information:

"Credit for the term 'bug' is generally given to the legendary US Rear Admiral Grace Hopper, a mathematics professor turned naval technologist, who found and removed ("debugged") a moth blocking a relay in Mark II, an early digital computer for calculating trajectory for naval guns."

by mallardtooxx

one day I was working on a sunday so I could get ready for a week long trip. I decided to take my son into work with me since I was going to be gone and he enjoys being @ daddies work. About 3 hours into the morning my son was in the middle of a massive army man battle on the floor. I thought "what the hell I am still a kid" and decided to play with him for a few minutes. My son "shot" me and I began an oscar award winning death scene. In my death theroes I flung my arms up and, much to mysuprise, heard a loud Bang. I had navigated myself into the phone system and then proceeded to COMPLETELY destroy it by pushing a cover into a power conduit and short the main box out. Needless to say I missed the first two days of the trip tryingto explain just how the "prowler" broke in and destroyed our phones to our vendor. It always goes over well when you have no phones on monday morning.
My son however thought it was pretty cool when the sparks flew!!

-duck

by road-dog

Well son, that wasn't an accident, it was a simulation of battlefield conditions. The smoke, flames, and stench of destruction are what you can expect in combat conditions.

Let this be a lesson to you that war isn't funny.

by mallardtooxx

No son that is not a fire but a deftly crafted simulation of fire and the results it can have if left unchecked.

So I suppose you are going to tell me then dad that is mom simulating the stop drop and roll technique while her hair burns off?

Yes son that is exactly what I am going to tell you, now go back to bed!!!

-duck

by tomsal

Hey MallardtooXX,

lol...got that one on tape by chance? How much for a copy? ;)

by lordinfidel

I was getting ready to deploy a whole slew of new machines. I worked on this image for about 2 days getting all of the software, permissions, etc installed and configured.

Went to clone the drive onto the rest of the machines.

So I opened up the first machine that was supposed to be cloned. Disconnected the cdrom's ide cable and plugged it into the master clone drive.
Started the process thru ghost...
Disk to Disk....

OMFG!!! I just overwrote the new image with the old image.

Iforgot to swap the ide cables. Since they were both masters on their own controller and the old drive was on IDE0. I needed to put the old drive on IDE1 and the master clone drive on IDE0.

Oh well, luckily I documented most of the install beyondwhat I normally do. But I held my head in shame and disbelief at what I had just did.

--------------------

Or for something even funnier.

I have some huge honkin bad boy servers, the compaq ML580. If anyone has seen these they are 12U andweigh 200 pounds easily.

Well I placed these in our brand new colo, and found that one day I needed to reboot it. Well being 25 miles away, obviously I would do this remotely. It just so happens that this particular server is just one of my bastard utility servers that I really don't care about. So I only plugged in 2 of the 3 power supplies.

Well, let's just say that after 10 minutes after the reboot and I could not reconnect, I figured something was wrong and a drive was in order. Turns out that the ML580 does not have a bios bypass function on errors. You have to press the F1 key to continue.

This wouldn't be so bad unto itself. But for about a month after that, every server I rebooted failed to come back up. 2 of my linux boxes, a couple of windows servers. It was like I had the touch of death.

by hal 9000

Foul up but I'll settle for my lattest foul up started about 3 weeks ago on a Sunday when a so called friend {Who I have now drowned} rang me up and said that his friends computer was showing a message of iminent HDD failure, what should he do? Being the nice person I am I told him to shut the thing down and bring it to me immediatly which he dully did and I got the same message so I disconected the HDD and ran a test on it only to come back at 100% so I figured that there must be something else going on an fairly normal situation. Anyway after a few hours of mucking around I asked has anything been added recently only to be told tha a LS120 dirve had just been put in it was the slave on the primary IDE line and the master jummper wasn't set correctly Easy right? Well that fix lasted all of 2 days before I recieved a phone call that it wasn't working again. Now this guy runns a small business & I mean really small 1 man band who only uses the computer on a Tuesday to put out an invoice so he can be paid at the end of the week so it was of some importance to him that the computer wasn't working on the Tuesday. I duly went down there a 40 Km trip only to find a dead HDD just goes to show don't always believe the testing results. Luckly I had saved the entire contents of his HDD to a file on my server so I brought him back to my place so he could do whatever he required to get paid and then I set about "Fixing" again his computer he had by this time had a talk to a friend and he wanted 2 HDD one to boot from and one to store data so in the event of something going wrong with Windows XP Pro in this case his data would be safe!

by hal 9000

At the same time I suggested that I fit an external modem for security reasons as his 6 yo daughted played games on the computer and it was probably not a good idea to have it open to a phone line while she was playing with it. He agreed and I set about rebuilding the computer to what he wanted but he did get a bit carried away as he asked prices on everything and started wanting a whole new computer built from the remains of his existing one I had to draw the line somewhere so I suggested thatI put a bit more Ram into the computer {About 4 times as much as it came with as there was not enough for XP}, the HDD and a external modem. Well that went well enough with a total reload and for a reduced price of the bits I sent him on his merry way only to get a phone call about 2 hours latter telling me that the thing was not working again. So down I went again only to find a totally trashed OS that was unrecoverable in any way short of doing a reload. I naturally asked wht had happened andhe told me all he had done was to try to install SP1, I figured that he must have done something wrong and I again took the computer away and reloaded it but this time I installed SP1 only to have the same thing happen again. Now this computer was built by someone else and came with XP Pro so I figured that the hardware should be OK but I ran a test just in case and everything showed up as all right so I set about trying to find the software fault. About 200 man hours latter and countless reloads I decided that there must be something wrong with his version of XP so I installed it onto a new HDD on one of my computers and then set up SP1 and everything worked just fine thankyou very much, so I was left with a computer that was susposely XPcomplient but would not work with XP.

by hal 9000

I then set about relaoding his computer with all his software and everything that he needed. Then I replaced the boot drive and everything else except the DVD drive as ti was master on the secondary IDE line and reloaded XP Pro and then installed SP1 only to have it fail again. This time I changed the DVD with the CDRW and did it all over again and it worked without problem simple I suspose the DVD was the problem but it would load XP now figure that one out? Anyway I replaced the DVD and setupthe computer but just in case I installed every update/patch that was available and once I was sure that it was working properly I returned it to him. # days latter he rang me complaining that the computer would not reconise the modem so I suggestedhe leave the modem turned on and reboot the computer, it worked fine so I suggested that he do this everytime he turned the computer on and then turn off the modem untill I could get down there and find out what was going on. Well today I spent another 6 hours fault finding only to come up with the conclusion that the Comm ports where being treated by XP as a possible security breach and if there was nothing present they where being deactivated on boot up. How to get around that one had me a bit stumped for a while so I suggested that I replace the internal modem and install the firewall that he had there. He wanted no part of this as he had already ripped out the firewall as it was causing him too many problems and it was Zone Alarm not the most diffictual firewall to work with.

by jardinier

Yes, Col. It brings to mind the Elvis hit "One of those days."

Have you ever had one of those days, boys
Have you ever had one of those days
When nothing goes right from morning to night
Have you ever had one of those days.

Thanks for your contribution to the "foul-up" discussion.

by hal 9000

The above was all my staff where coming in and having a look to see what I was doing only to offer their opinion but when I offered them the job they all naturally declined {I can't understand why}. But when I finally got it working they all wanted to know what I did and how I found the problem, so the only good thing to come out of it is that I am now not asnswereing the phone and have burried myself in one of those projects that you always put off to another day {Debugging some Visual Basic} that some one wrote and dosn't work, at least my staff know when to leave me alone as at the moment I'm likely to throw something heavy at them if I'm disturbed {thats not in any way implying that I'm not certifable anyway} Like a Server or somethingas big. It at least leaves me free to do what I want without interuption and everybody is too scared to pass a phone call on to me after this guy rang up earlier this week and I threw a MS book on writing secure code at one of them when they came into gloat and tell me that he was on the phone again.

Actually M$ is quite funny as they consistently ask just why I list myself as Alledged Boss on my original Microsoft Partners application form.

by hal 9000

Anyway the end of it all was he has now agreed to turn on the modem always before he boots the computer as it works just fine this way even if the modem is turned off while he's working and then decides to go online and turns the modem back on. The modem supplier who offers a 24/7 help line only coment was to flash the BIOS then flash the Firmware to the latest version and download the latest version of their modem driver, well I'd already flashed the BIOS, the firmware version was the latest one available and the same went for the driver so I was left with no where to go except change the M'Board. While the solutions not perfect it works and I at least have learnt never to believe that any of the oppersisition actually knows what their doing as the M'board is not XP compliant even though ASUS has a audio XP driver available it will work fine if you don't put on any patches. This just goes to shaow "NO GOOD DEED EVER GOES UNPUNISHED", I didn't even cover the cost of my petrol on this job and I'm scared to answer the phone in case it him again. Now whats got me worried is what will come next. By the way I had to return a monitor to the Queensland distributors repaired today and it was the company who made this computer so I don't hold out much hope of it being repaired and it's only a 21" CRT monitor so I'll probable for a quite life just buy a new one there only a little over $2,000.00 AU at my cost.

Has anyone got a better one than this?

by hal 9000

While I( was entering in the above lot my computer did a live update from MS and now is doing the same thing that I have just spent so much time trying to fix on another computer and for the second time in 3 days I can't get my E-Mail either. I swear to God I'm going to throw everything MS into the rubbish bin and switch over to Linux and never again touch anything from MS. But it just might be that my ISP has a problem with their Mail server, however the modem bit is getting to me now.

by jereg

We had installed new servers on our network and I was moving user data directories from their old server to the new one. A boring process, but it was copy, then delete the old directory. I had done about 100, and came to a manager with a rather large directory with very sensitive data. I promptly deleted it. When that was done, I realised that I had forgottn to copy it. Of course, sitting in the corner, nobody knew what I had done, and fortunatly, the user was out of the building that day. I quickly got last night's backup tape, and restored the directory. Then I moved it. Lesson learned, attention payed.

by hal 9000

Screw Ups is we learn far more from them than when everything is going well don't we?

This discussion is closed to new replies.