Got called once on Monday this week and then on Tuesday. A user in one of the trailers had problems accessing the server, which is a domain controller, file server, and application server.
On both occasions the client had restarted the server before calling me (I’m not on staff, I’m a consultant) hoping that this would solve the problem.
When that failed on both occasions, they decided to call me. On both occasions all I had to do was log the user off and log back in… problem solved. I told the client that the next time it happened they shouldn’t restart the server, that they should call me first thing.
Yesterday, Thursday, I was called again. This time the server wasn’t restarted and so I could troubleshoot. This time, several PCs were affected, including some on the main building.
Right away I realized that the problem had nothing to do with the slow Cat5 connection (less than 10Mps bandwidth) linking the main building with the other two trailers.
The messages on the Windows XP workstations spoke of security violations, and the inability to contact a domain server, but the server was up and running fine. In fact, a couple months ago I had installed a second NIC and enabled DNS round-robin (a crude but effective form of load balancing, which I decided to implement since Win2K Server only has load balancing in the Advanced Server version).
I checked the event logs on one of the affected PCs, as well as the server, there were no security violations. I decided to ping one of the network cards on the server, so I pinged X.X.X.4 (where X is an octet that I don?t want you to know). There was 100% packet loss, in other words, no reply. I decided to ping the other NIC card, X.X.X.5 and I got a response, 0% packet loss. Something was wrong with the first NIC.
I went to the server and checked the status; the first NIC was not registering any packet transmission activity, even though it was shown in the system tray as connected.
I have PC Anywhere running, which allows me to log into the server from one of the remote offices, so I began to think that maybe PC Anywhere had locked up the connection. I closed PC Anywhere, but that didn?t help. After all, PC Anywhere was listening on both NICs, why would it lock up only one?
I disabled the NIC and re-enabled it, packet transmission began on the NIC and everyone could now connect to the server without any error messages.
I promptly went out and got a generic $14.00 NIC from a popular retail store. That afternoon I ran a diagnostic on the NIC when everyone had left the office. The NIC failed the twister loop back test the first time, on the second try it passed all tests. On the third try it failed the Ethernet loop-back test. On the fourth and fifth try it passed all tests. On the sixth try it failed both the twister and Ethernet loop-back tests. I replaced the NETGEAR NIC (thanks to NETGEAR for their diagnostic tools, and solid documentation)generic NIC. Windows didn?t even detect new hardware. I had to take the NIC back out and return it to the store last night. Instead I got a LINKSYS NIC which I installed this morning, and everything is working fine again.
The moral of this story is:
1. Customers should not try to cut corners by doing stuff like restarting the servers. In this case, I?m going to bill the customer for those two occasions that they restarted the server and I had to drive out, only to log the user out and log them back in. Had they not restarted the server I would have figured out the problem the first time.
2. Regardless of how many messages you get from a machine, always use good troubleshooting methodology, PING, TRACERT etc.
3. The most important, don?t use cheap hardware, it costs you in the long run.
