Questions

A very strange problem concerning PoE, switches, and point to point...

Tags:
+
0 Votes
Locked

A very strange problem concerning PoE, switches, and point to point...

kmthom
I challenge ANYONE to figure this out...

Before I begin my explanation of the problem, please understand that at this time the network\internet WORKS. However, the method in which I employed to get it to work is very....strange. In short, I am wondering how I have connectivity at all at this point. Here we go...

I was called out to one of our buildings the other day after they had just been without power for around 5 hours. Staff inside had no internet connection, after the power was restored, of course. I first checked outside; we employ a wireless point to point system offered by Airaya. (www.airaya.com) So my first inclination was that the antenna had been knocked out of alignment. This proved to be true, and after adjusting the antenna a little I was able to plug into the bridge of the antenna and access the internet.

Down in the wiring closet, things were not so simple as the rest of the building was still without internet. The bridge I spoke of resides in the attic, so we have a PoE connection carrying the data and power up to the bridge through Cat5. This was put in place so that trips to the attic would not be necessary to power cycle the bridge - it could easily be done from the closet. The cable coming from the attic first hits a small patch panel (don't ask me why, i didn't install it, and to me looks like a waste of patch cables) and from that patch panel the PoE injector hangs, then the Cat5 coming off the injector connects to a Netgear ProSafe 24 Port 10/100 Switch (Model: JFS524) Link: "http://support.netgear.com/product/JFS524" - From the switch we have a convoluted mess of cabling that in some cases goes back to the patch panel, others go out to their respective places on the network, mostly workstations and printers. Don't worry about the mess of cabling because it becomes obsolete when I explain the problem....in short, the 24-port feeds the whole network from the internet coming in on port 1.

The problem is, when I plug my laptop into the LAN port of the PoE injector, I get internet just fine. When I plug said PoE into the switch, then plug my laptop into the switch, i get nothing.

Now, since I can get the internet from the interface on the PoE, I am left to believe the network hardware up to that point is fine. This includes the antenna, bridge, PoE, all wires in between, and the hardware back at the transmitter tower...the internet works.

Any troubleshooter worth his or her salt would now argue, "It's the switch stupid!! Get another one!!" I'm way ahead of you. In fact I tried 3... Same problem, but, same brand, same model...JFS524. I hardly think that matters though, stay tuned.

In order to simplify the problem, when trying the new switches (by new I mean they just came out of the box and plastic) I eliminated ALL other connections, besides the Cat5 from the PoE injector (the internet) and my laptop. Still nothing. I got the internet cable plugged into one port, my laptop plugged in another and I got nothing...on three different switches. When I eliminate the switch, and plug directly into the LAN port on the PoE injector...TADAA!!! Internet.

Remember, these are dumb (smart?) switches. NO configurations.

I have never come across this...but I don't have much experience with PoE - so I was blaming it the whole time.

HOW I FIXED IT:

I found a small 5 port switch we had sitting around (http://homesupport.cisco.com/en-us/wireless/lbc/ezxs55W). I brought it over to the closet, connected the "Internet" line to it, connected my laptop to a free port on the small switch, and TADAA!!! Internet.

Ok, the easy thing would be to then connect all the lines to the little guy, and be done with it. Not so simple. This little device does not have enough ports for everyone in the building. So, in a last ditch effort, and out of sheer morbid curiosity, I link the 2 switches together....TADAA!!! Internet. For EVERYONE.

Yes. Somehow attaching the internet line to the little switch, then the little switch to the big switch fixed all internet connectivity for the entire building.

My question is HOW? All signs seem to point at the 24-port being bad, however that is obviously not the case. I also thought DHCP was to blame, but the IPs on each machine are static. The DNS server was also checked, rebooted then rechecked...no problems.

If nothing else, I hope this post at least helps ONE other person...and if somebody knows WHY this is happening or has run into it before I would love to hear it.

-=K=-
  • +
    0 Votes
    Rob Kuhn

    While not the same as what you went through, I had to employ something similar.

    I had four T-1 lines which we bonded by connecting into the back of our Netvanta router.

    At the firewall, the downlink was 5-6 Mbps but the uplink was 3-4 Mbps.

    We then removed the firewall and connected a laptop directly to the router and got the same result. So we could rule out the firewall for now.

    Thinking that one (or all) of the T-1s might have a problem we connected each one to the router one at a time. Each line measured 1.5 Mbps up and down.

    We contacted Netvanta support who examined our config file and logs. He then suggested putting a switch between the router and firewall.

    When we did that we got the full 6 Mbps up and down. The support person said that it sounded like a duplexing problem(?) and vy putting a switch inbetween "balanced" it out.

    Ths was back in 2006 and we ran it that way until 2008 when we dropped in a DS-3 (45 Mbps) circuit and retired the T-1's.

    We never did learn what the real problem was. All we knew was that it worked. :)

    +
    0 Votes
    kmthom

    Very similar indeed! It was frustrating when I was looking at this thing wondering, "Ok, this SHOULD work..."

    Duplexing very well could be the problem. I guess I'll find out this fall when I rewire the closet. :-)

    -=K=-

    +
    0 Votes
    Rob Kuhn

    This might be moot but have you taken the time to sketch/diagram the configuration as it is right now? I find that drawing it out on paper sometimes help with the troubleshooting. And when it comes to *weird* setup/configurations I like to put a copy with the equipment or at least somewhere onsite in case another IT person has to troubleshoot and I'm not readily available.

    Even if you do plan to rewire and redo the entire infrastructure it's still worth documenting the setup as it stands now just in the off chance you have to put it back or if it should fail before you have a chance to redo everything. :)

    FWIW, I'm pretty big into documenting things. Part of it has to do with the industry that I'm in and because early on in my IT career I had no one that I could really turn to so I had to take notes on everything I also had a boss who was a private pilot and he was big into checklists. :) That's stuck with me over the 20 years and I burned that trait into all that worked for me over the years :)

    Best of luck and great question!!

    +
    0 Votes
    kmthom

    Since I took this position (my first IT position btw) I have documented and diagrammed most of the important areas of the network. I am the lone IT guy on a 110 acre campus, with around 10 buildings utilizing wireless point-to-point, and over 100 network devices. I received 2 hours of "training" from the company that maintained the network before I got here, then the guy hit the road.

    I wholeheartedly agree on documenting and diagramming. When I first got here, I was disappointed to find the previous company provided very little in terms of documentation on the current network setup. I had a list of static IP's and their respective clients (no active directory, nor DHCP for most of the network - but that is a different story.), but little else.

    I turned to SpiceWorks.com, ran the client, and scanned the network. It does not work very well simply because the network is a mess, but it really helped me find out just how large the network was.

    -=K=-

    +
    3 Votes
    robo_dev

    The first question is, when you 'get nothing' on your laptop, is the link light on?

    If the light is on, the next question would be whether the switch-to-PC connection connected at 100-full, 100-half duplex, 10 full, or 10-half duplex. A duplex mismatch will cause communication to fail...sometimes connectivity WILL work if it's mismatched, but performance will suffer.

    Also, the switch has 'auto uplink' which means if you plug a straight-thru into it, it will automatically goto crossover mode. If you used a patch cable that was a crossover cable, it would work for the POE device, but not for your laptop.

    The POE adapter also has to negotiate link speed and duplex, and it's getting it right since when you plug into it, it works.

    The other issue is if the switch has STP (spanning tree protocol), there is a delay when connecting....sometimes up to 20 seconds. Also, if there were a network loop. spannning tree would 'block' the port. Note that you CAN create a network loop with a wireless bridge.

    +
    0 Votes
    kmthom

    Interesting ideas.

    The connection speed issue didn't cross my mind - good idea there. The next time I find myself in that closet I will check it out. For now, it works so I am leaving it alone until we rewire everything.

    I thought of the straight-thru and cross-over scenarios, but did not investigate too much because the system worked fine before the power got knocked out. From memory, I think all of the cables were straight-thru, but don't quote me on that.

    I can't find any documentation on weather or not the 24-port has STP. The spec sheet does not list it, but who knows. This is another theory which may be helpful though. The closet is a convoluted mess that was installed before I was hired; so, the possibility of there being a loop someplace is entirely likely. But then again, I still could not connect after I bypassed all of the other connections. I can't wait to get in there and rip it all out. >:-)

    Thank you for your insights!

    -=K=-

    +
    0 Votes
    CG IT

    if the PoE injector isn't supposed to power anything. I'm assuming that the little 5 power switch isn't being powered by the PoE injector and that you found a wall outlet to plug it's power cable into. Also assuming the PoE injector isn't powering any of the 24 port distribution switches.

    Since the bridge works and that is getting it's power from a PoE switch [or is it getting power from an injector?] as long as the injector isn't supposed to power anything, the whole shebang should work without your 5 port switch. If the injector is supposed to power any of the distribution switches and that's where the network connection fails, I'd look at the PoE injector given that you had a power outage and after restore of power the problem started.

    +
    0 Votes
    kmthom

    The PoE injector in the closet is powering the point-to-point bridge device up in the attic. Eventually, we are going to move everything down from the attic into the closet to simplify things. At that time the PoE injector will earn its rightful place in my parts bin...

    I'd love to run everything off the little 5 port switch, but my ignorance of the "inner-workings" of the building entire network prevents me from yanking cables and doing much at this time. Plus, I need more than 5 ports to satisfy everyone.

    +
    0 Votes
    oliverb2012

    I'd take a guess that the main switch is failing to auto-negotiate, I have a printer that will only connect via a "Dumb" 10baseT hub. I can connect the hub to a switch just fine.

    +
    0 Votes
    hrisan

    Hi, the connection bridge to switch, was it with cross-over cable or straight? If it is not cross-over this may cause problems although it should work with straight through cables. We had heaps of problems between 2 sites connected with P2P link and at the end it was a deplexing issue cased by the fact that it was not used cross-over cable.

  • +
    0 Votes
    Rob Kuhn

    While not the same as what you went through, I had to employ something similar.

    I had four T-1 lines which we bonded by connecting into the back of our Netvanta router.

    At the firewall, the downlink was 5-6 Mbps but the uplink was 3-4 Mbps.

    We then removed the firewall and connected a laptop directly to the router and got the same result. So we could rule out the firewall for now.

    Thinking that one (or all) of the T-1s might have a problem we connected each one to the router one at a time. Each line measured 1.5 Mbps up and down.

    We contacted Netvanta support who examined our config file and logs. He then suggested putting a switch between the router and firewall.

    When we did that we got the full 6 Mbps up and down. The support person said that it sounded like a duplexing problem(?) and vy putting a switch inbetween "balanced" it out.

    Ths was back in 2006 and we ran it that way until 2008 when we dropped in a DS-3 (45 Mbps) circuit and retired the T-1's.

    We never did learn what the real problem was. All we knew was that it worked. :)

    +
    0 Votes
    kmthom

    Very similar indeed! It was frustrating when I was looking at this thing wondering, "Ok, this SHOULD work..."

    Duplexing very well could be the problem. I guess I'll find out this fall when I rewire the closet. :-)

    -=K=-

    +
    0 Votes
    Rob Kuhn

    This might be moot but have you taken the time to sketch/diagram the configuration as it is right now? I find that drawing it out on paper sometimes help with the troubleshooting. And when it comes to *weird* setup/configurations I like to put a copy with the equipment or at least somewhere onsite in case another IT person has to troubleshoot and I'm not readily available.

    Even if you do plan to rewire and redo the entire infrastructure it's still worth documenting the setup as it stands now just in the off chance you have to put it back or if it should fail before you have a chance to redo everything. :)

    FWIW, I'm pretty big into documenting things. Part of it has to do with the industry that I'm in and because early on in my IT career I had no one that I could really turn to so I had to take notes on everything I also had a boss who was a private pilot and he was big into checklists. :) That's stuck with me over the 20 years and I burned that trait into all that worked for me over the years :)

    Best of luck and great question!!

    +
    0 Votes
    kmthom

    Since I took this position (my first IT position btw) I have documented and diagrammed most of the important areas of the network. I am the lone IT guy on a 110 acre campus, with around 10 buildings utilizing wireless point-to-point, and over 100 network devices. I received 2 hours of "training" from the company that maintained the network before I got here, then the guy hit the road.

    I wholeheartedly agree on documenting and diagramming. When I first got here, I was disappointed to find the previous company provided very little in terms of documentation on the current network setup. I had a list of static IP's and their respective clients (no active directory, nor DHCP for most of the network - but that is a different story.), but little else.

    I turned to SpiceWorks.com, ran the client, and scanned the network. It does not work very well simply because the network is a mess, but it really helped me find out just how large the network was.

    -=K=-

    +
    3 Votes
    robo_dev

    The first question is, when you 'get nothing' on your laptop, is the link light on?

    If the light is on, the next question would be whether the switch-to-PC connection connected at 100-full, 100-half duplex, 10 full, or 10-half duplex. A duplex mismatch will cause communication to fail...sometimes connectivity WILL work if it's mismatched, but performance will suffer.

    Also, the switch has 'auto uplink' which means if you plug a straight-thru into it, it will automatically goto crossover mode. If you used a patch cable that was a crossover cable, it would work for the POE device, but not for your laptop.

    The POE adapter also has to negotiate link speed and duplex, and it's getting it right since when you plug into it, it works.

    The other issue is if the switch has STP (spanning tree protocol), there is a delay when connecting....sometimes up to 20 seconds. Also, if there were a network loop. spannning tree would 'block' the port. Note that you CAN create a network loop with a wireless bridge.

    +
    0 Votes
    kmthom

    Interesting ideas.

    The connection speed issue didn't cross my mind - good idea there. The next time I find myself in that closet I will check it out. For now, it works so I am leaving it alone until we rewire everything.

    I thought of the straight-thru and cross-over scenarios, but did not investigate too much because the system worked fine before the power got knocked out. From memory, I think all of the cables were straight-thru, but don't quote me on that.

    I can't find any documentation on weather or not the 24-port has STP. The spec sheet does not list it, but who knows. This is another theory which may be helpful though. The closet is a convoluted mess that was installed before I was hired; so, the possibility of there being a loop someplace is entirely likely. But then again, I still could not connect after I bypassed all of the other connections. I can't wait to get in there and rip it all out. >:-)

    Thank you for your insights!

    -=K=-

    +
    0 Votes
    CG IT

    if the PoE injector isn't supposed to power anything. I'm assuming that the little 5 power switch isn't being powered by the PoE injector and that you found a wall outlet to plug it's power cable into. Also assuming the PoE injector isn't powering any of the 24 port distribution switches.

    Since the bridge works and that is getting it's power from a PoE switch [or is it getting power from an injector?] as long as the injector isn't supposed to power anything, the whole shebang should work without your 5 port switch. If the injector is supposed to power any of the distribution switches and that's where the network connection fails, I'd look at the PoE injector given that you had a power outage and after restore of power the problem started.

    +
    0 Votes
    kmthom

    The PoE injector in the closet is powering the point-to-point bridge device up in the attic. Eventually, we are going to move everything down from the attic into the closet to simplify things. At that time the PoE injector will earn its rightful place in my parts bin...

    I'd love to run everything off the little 5 port switch, but my ignorance of the "inner-workings" of the building entire network prevents me from yanking cables and doing much at this time. Plus, I need more than 5 ports to satisfy everyone.

    +
    0 Votes
    oliverb2012

    I'd take a guess that the main switch is failing to auto-negotiate, I have a printer that will only connect via a "Dumb" 10baseT hub. I can connect the hub to a switch just fine.

    +
    0 Votes
    hrisan

    Hi, the connection bridge to switch, was it with cross-over cable or straight? If it is not cross-over this may cause problems although it should work with straight through cables. We had heaps of problems between 2 sites connected with P2P link and at the end it was a deplexing issue cased by the fact that it was not used cross-over cable.