Some years back I was working on a high security gateway, on of the junior admins on the gateway. I was designated as the level 2 and level 3 tech for all help desk tickets relating to the gateway. My job was to investigate the issue, and fix if I can. If not get one of the senior admins to fix. 95% of problems I was able to resolve.
Well, one day we got this ticket that had already been around the world, every single tech section in the company had dealt with it at one point, still unfixed. Someone had decides it was a gateway issue, and sent it to me.
The problem: Intermittently mails would arrive (both inward and outgoing) scrambled. Dozens of people were affected, but only the odd mail.
I checked everything and all looked good. Sent out an alert to all the people who used our gateway, if you get a scrambled email, contact me immediately. I reset the mail servers to keep a copy of all mail, they would, but only for a short time.
Over the coming weeks, I got five reports and checked them out. I had the evidence the scrambled incoming mail was good when it went through the gateway, thus scrambled later, and the scrambled outgoing was scrambled before reaching us. It wasn't a gateway problem.
However, every other section had done checks and proved it wasn't theirs. After getting clearance from my manager, I spent time looking into it. After some time, I was able to say it wasn't a problem with the hardware, this was a software problem.
I mapped where all the affected individual clients were physically, all over the country, from one end of Australia to the other. I mapped where they were on the logical network, they were in various different branches and divisions.
Finally, in desperation, I sent emails to each reported client using a trace route program. When I got all the reports back from the 80 clients, I found the only thing they had in common was the gateway mail server. The network had some dozen mail servers throughout it. Closer examination showed all affected clients were behind one of three regional mail servers. Call all the servers MS01 to MS12, all the scrambled messages went through MS03, MS04, or MS08.
More research, every mail server within the network, except the gateway(we used Sendmail) was using the same mail server software. However, MS02, MS04, and MS08 were using version 4 and the rest were using version 5. I checked, vs 4 used uucp as default, while vs 5 used smtp as default. Hmmm. Vs 4 can be set to use smtp and they were.
I checked the date stamps the mails went through the servers, and the date stamps of when the servers were reset for maintenance. each faulty mail was the very first following a reset. Hmmm.
I sent the mail server help desk team the ticket with the details of my research and a note saying the problems was due to the mail servers expecting uucp messages and they dealt with the first after a reset as uucp, this was a protocol handshake issue with the vs 4 of the software.
They rang me told me I was wrong, nothing was wrong with their servers. They insisted it was a gateway issue. That what i said was happening was impossible. they refused to spend $8,000 upgrading the hardware and software on the three servers mentioned.
I got some severe abuse from them over it, yet not one checked my work - these guys were supposed to be very senior help desk techs, paid more than double what I was.
I stood my ground, and ended up standing in the unemployment queue.
Seven months later, I got told by a friend who still worked there, the company spent $25,000 getting experts from the USA out, and they couldn't find the problem. My friend showed them my reports and work, they checked it, ran tests then wrote up a new FAQ for their software reporting this new, previously unknown problem with the vs 4 software.
Since then I was asked to go back to work for that company - now way, not unless they sack the fools who refused to believe me because I didn't have the same quals as they.
But was I mad at those 'Help Desk level 3' idiots. They had the special training and knew nothing about it, so it couldn't possibly be the problem.