IT Employment

Ask something
Discussion Locked

When IT admins make mistakes...

0 helpful
2 replies

by oliviu

Closed to new repliesThis discussion is closed. It stays readable, but it takes no new replies.

I was recently in a very unpleasant situation.
I made a mistake while improving a customers SCO server and his server was no longer able to boot properly, offering the "panic..." message.
He has critical data and applications there and the contact person is not a technical one so I had to dictate character-by-character commands and instructions on phone in order to have the problem fixed. "Happily" this happened after 5p.m. and the system was back next day before noon (I had to send him a couple of programs by FedEx).
Normal, the customer informed the general manager of my company and I had a ?nice? discussion with him. Only the argument that their data was never in danger because I was very carefully with it and the support from my college was able to bring the discussion to a normal way. Now I?m still waiting for a possible punishment and I?m reviewing my resume.

I?m curious to hear some of your experience(s) related to small or big mistakes you did and what kind of punishments/rewards followed. What you consider to be a good policy in dealing with this kind of problems?

P.S. 3 years ago I herd about an Oracle DBA that dropped one of the company databases. He was fired and in the same day he was hired by the company located next floor with a higher salary. So this could be considered a reward for a mistake. Maybe is only an urban legend, but what about if...

2 replies

by g.m.bakker (cne, self employed)

Hmmm,
I guess the client has 2 UPS's, redundant powersupplies, 1 spare mirrored server, never skipped a backup, always cleans his streamer and saves backups in a fault 100 miles away...If not he shouldn't whine...
Hardware fails, so systems go down, just too bad you tinkered a wrong 'bit', as long as you make backups before tinkering. I accidently crashed a server and we were down for a day (office hours) and 180 people were picking noses. All I did was stop and start a printserver on a Novell 3.12 server after changing a printer queue, the server which abended, and had to be powered down...Unfortunately had no reset switch, so we had to power down. On applying power the 9 month old SCSI harddisk died...Well, I did all the bad stuff which at that moment was all there was left to do, and unfortunately it back fired. So if your tinkering was to enhance performance without upgrading expensively, or solving a problem elsewhere, it's just too bad, shit happens...
Also...how about a NT server, with all those extra options and choices OK CANCEL CLOSE...any screen/Window you view should be closed with a cancel...unless you want to change something and risk rebooting. And in an environment where several admins are taking care of a server park, it's never sure who did the deed and pressed okay..instead of cancel..and didn't reboot resulting in a sudden crash of the NT server a fortnight later!

by dlongan

I have had some very interesting events take place in my 18+ years in the IT industry. But never once have I been fired, demoted or such.

I always explain that things can go wrong, and that's the big reason to have backups...

Couple of quick events:

Company: Automotive Manufacture
Servicing a power distribution system.
Testing the e-stop switch (PDS is suppose to be in "test mode" drop electrical power to the entire computer room.

Company: Automotive Manufacture
Scoping the signals on a backplane, accidently drag scope probe over backplane pins and smoked all options in the cabinet.

Company: Defense contractor
Servicing a 9 track tape drive.
installed power board incorrectly
smoked entire unit, in front of the customer!

Company: Manufacture
Servicing owner's pc from home, wiped out entire hard drive. no backups

This discussion is closed to new replies.