"Building 2 - has 3 switches and a CAT5 connection into the Main Building switch."
A "discussion" (translation: argument) that I sometimes have with customers is about the merits of running a fiber-optic connection between buildings, instead of copper. Yes, fiber costs a little more, but there are two HUGE benefits:
#1 - lightning and surge protection. Anytime copper goes between buildings, there's an increased chance for transients, even if the cable is underground. Fiber-optic is not subject to it. Benefit: increased safety, minimal cost. In fact, the difference between the cost of fiber and the cost of TWO Cat5 surge protectors (one for each end) is often minimal.
#2 - ground loops. Different buildings have different electrical grounds. Sometimes there is a measurable potential between ground in one building and ground in another building. When you use copper (like cat5) between buildings, you connect the electrical grounds together. Sometimes, there's enough potential to wreak havoc. Try measuring it with a voltmeter - run a regular extension cord from one building to the other, and then measure from "ground" on one (the 3rd prong on the plug) to "ground" in the other building, like a conduit. (BE CAREFUL! I've seen cases where the ground loop was 100 volts or more! DON'T TOUCH! POSSIBLE SHOCK HAZARD!) If it's more than five volts or so, it could be a problem for computers. Again, Fiber-optic is not electrical, so it's not subject to ground loops. Un-problem.
Ground loops cause a "hum" (longitudinal imbalance) on the line. The receivers in the switch can compensate to some degree, but there's a limit. I'm guessing the bad switch may have shorted the chassis ground to the balanced line in the Cat5, and pushed the signal distortion into the other switch.
Yes, switches with one or more fiber ports are more expensive than Cat5-only. If the customer is really that cheap, there are "media converters" readily available, that have Cat5 on one side and Fiber on the other. They're less than $100 a pair on ebay. Then just plug the Cat5 side to the switch on either end, just like before. Each building's media converter is plugged into that building's power, so there's no ground loop.
And BTW, the same thought applies for buildings with more than one electrical service: Ground loops can be a problem, and Fiber-optic is an easy way to avoid it.
As long as it's just a straight-run point-to-point fiber, the connectors are very easy to install. Then plug in the fiber, plug in the Cat5 and it's ready to go.
And of course, fiber has the potential for FAR more bandwidth than Cat5 - which may become another important payback as the customer grows. Change out the switches and go Gigabit on the fiber, and Gigabit directly into the server.
BTW, my curiosity: Why does server have 2 NICs plugged into the switch? Would it make sense to plug one NIC into the Building One switch, and plug the other NIC into the Building Two switch? That would make subnetting easy!