Networking

Unmanaged switch horror stories

The use of unmanaged switches irks many network administrators and makes trackbacks very difficult. In this blog post, IT pro Rick Vanover shares a few stories of what users can do to a network.

In reading last week's 12 things to know when troubleshooting your network post, a particular chord was struck with me in the physical layer verification section of the document. That chord is the ongoing battle with unmanaged switches in an enterprise network. The practice of users putting up unmanaged switches and hubs is a pet peeve of many network administrators, as well as generally forbidden by policy though not always enforced by the managed switches.

Knowledgeable workers are generally smart enough to obtain a network switch and uplink it to the company network, and the onus is on the network administrator to correct any subsequent issues that arise. While this is usually harmless, it can sometimes go awry. Here are some situations that I’ve seen in my time:

  • Using the built-in switch of a router as a switch. This is the good old rogue DHCP server battle with a new twist. The user was using a DSL router that was used in the branch office as a switch on their desk. Here is the crazy part, the user had uplinked and simply "got used" to getting the correct DHCP address or the closed branch office’s network -- that went to nowhere. The DSL router even had a label of the city that the branch office was located on the top -- and that was a thousand miles away.
  • Uplink loop. Having a switch uplinked twice to the network or to another switch two times makes for a pretty graph of network utilization, yet it becomes incredibly inefficient at moving packets.
  • Broadband router and switch. One user had been using a wireless broadband router with the Internet connection of the data card because the Internet content filtering was annoying.

These accounts come from remote sites that don’t have an IT staff footprint locally, and in many accounts the users truly think that they are on their own. The stories are out there, and surely you have had some sort of run-in with a remote switch that has stirred you up. Please share your stories below.

About

Rick Vanover is a software strategy specialist for Veeam Software, based in Columbus, Ohio. Rick has years of IT experience and focuses on virtualization, Windows-based server administration, and system hardware.

38 comments
mlaxton
mlaxton

What do you all do in the cases where there is only one drop in an office and a user needs connections for a pc and a networked printer? In the past I've just had to use a small unmanaged switch to get everything on the network. Is this a bad idea when you can't run any more drops without major hassles? Just wanted to get some more opinions. Thanks!

Zoidburg
Zoidburg

I had an employee once connect a home Router to the network to he could have wireless access with a laptop. Needless to say IPs suddenly started poping up on PCs and were not in the ranges of our routers. The issue was quickly resolved and DHCP was killed on the device (the wireless need was actually justifiable)

mnleblanc
mnleblanc

I manage a small k12 LAN at a regional high school. I have, over the last four years, replaced most of our unmanaged switches with Cisco equipment. It is really quite amazing how quickly the simple looping of a cable in an unmanaged desktop switch can bring a 400 node network to its knees! There are still times when I just can't avoid dropping a desktop switch somewhere for expedience, but linking them to managed devices allows some mitigation through storm control.

randy_scadden
randy_scadden

Well I have a couple of horor stories. In the last company that I worked for when I walked in it was a disaster. Every port in the entire company was on an unmanaged switch and there where just major issues. The first scenario was we had a call center with about 12 to 15 reps who where all using VOIP softphones. Whenever there where more than say 4 or 5 reps on the phone at the sametime all of the sudden the phones would just have horrible echo. Well when I saw that the switch it was going into was an off the shelf 24-port Hawking Technology unmanaged nightmare I immediately went into the CIO and told him that the issue with the echoing was related to the switch. Now mind you the CIO was in his role because he had been with the company for years and had grown into the position. On top of that he had no formal background in IT or corporate / enterprise infrastructure support. His response to me is that the echo couldn't be caused by the switch as it had more than adequeate bandwidth and capacity to handle all of the VOIP traffic. Well after about 2 months of both myself and our outside consultant hammering away at him telling him that his echo problem on the VOIP phones would go away. Well he finally took the jump and we convinced him to go out and purchase some used Cisco Managed switches. Well I deployed that switch in the morning and literally by noon I had the director of the call center so ecstatic that he was now able to have all of his reps on the phone at once without any echo. The second was a situation where someone plugged in a good old Linksys Cable / DSL Router into the network and it started handing out addresses. I ended up driving almost 400 miles that day just to go up and figure out that someone had installed a rogue wireless router onto our network and then I confinscated it. But it was nice once we migrated everyone to managed ports it definetly sped things up considerably.

Duluth Networker
Duluth Networker

Three times outside vendor's technicians have brought in their own temporary mini switches and attached them to our network as part of their troubleshooting and support. That's OK if they know what they're doing, but when they have too many cables, or cables that are too long, sometimes they plug both ends of their cable into their mini switch. If the mini switch is attached to our network, it creates a loop that is not caught by STP because the Listen/Learn of BPDU's is already completed--even more so with FastStart enabled on the edge port. Fortunately, we have Nortel switches, and SLPP (Simple Loop Protection Protocol) can shut down the offending port to the mini switch. SMLT (Split Multi Link Trunks) ports from the edge to the core also have SLPP enabled, so the problem is automatically either shut down at the offending port, or at the closet level. Both applications of SLPP prevent propagating the loop through the Distribution & Core gear and having multiple closets/users shut down. Repeated requests to the vendor to prevent this type of activity from happening see to have helped. Especially when the technician becomes educated and understands the impact of their looping actions.

daileyml
daileyml

Unfortunately situations such as these occur far too often in a corporate network environment. I'm not sure why this "lesson learned" is still being repeated over and over in networks everywhere. Network administration best practices dictate that unused ports are either administratively shut down at the switch or safeguards are used such as Port Security to specify or limit the number of MAC addresses than can appear on a port. Coupled with common Spanning Tree Protocol options such as BPDU Guard the use of unauthorized switches can be eliminated almost entirely in the majority of network environments. The introduction of unauthorized access points (such as broadband SOHO routers w/ built in wireless) can be dealt with using similar means, but I view this as more of an administrative issue than a technical one. If company policy dictates "no use of unauthorized wireless devices" and they are found on the network this becomes an HR issue, not an IT issue. -Mike D http://www.daileymuse.com

me19562
me19562

Most of those issues could be avoid with the appropriate switch configuration. For example in Cisco switches every port that is not connected to a switch or router should be configure with spanning-tree portfast and spanning-tree bpduguard enable. The use of spanning-tree portfast make the interface to change directly from a blocking state to a forwarding state without making the intermediate spanning-tree state changes. This normally fix the issues when a host is not getting an ip address from dhcp during boot. The use of spanning-tree bpduguard make the interface configuration command to put an interface in the error-disabled state when it receives a bridge protocol data unit (BPDU). Basically, if someone connects a switch to a port with bpduguard enable the port will be disable. The Uplink loop shouldn't be an issue in enterprise switches(assuming everything is configure right) since spanning tree will calculate the cost for every uplink and determine the root and alternate ports. The root port will be in Forwarding state and the alternate will be in Blocking state which will avoid any loop issues.

sidekick
sidekick

I had a call from a client that they were having network issues. They only had one switch. When I started investigating it, I was following a network cable coming out of the switch and it went right back in. Unplugging it fixed the problem. Appearently, it was an extra cable plugged in. When someone saw the free end, they thought it needed to be plugged in somewhere and stuck it in the switch. Silly users.

erilaurence
erilaurence

i want to know more about how this uplink loop happens. im new to networking and i want to avoid some mistakes or identify one when i see it. thanks

Deadly Ernest
Deadly Ernest

that much more money and provides a much better control of the situation, much more so with a remote office. Personally, for a remote small office, I think a reasonable small business or home router would be ideal. that way, you have remote management and can also expand by adding switches later - if need be.

rndmgr
rndmgr

Excellent point! Thanks! I have the same question. Based on what I'm reading here thusfar, a properly designed and configured enterprise network should allow for cubicle switches. Perhaps from an approved list of switch SKUs and with some IT oversight and authority. E.g., a rogue DHCP server will only effect the VLAN or subnet to which it gets connected. Other scenarios would cause that port to disable itself. These are acceptable risks to me compared to the cost of additional drops or adding managed switches. I'd like to hear about the dangers or down side of this as compared to the cost of more drops into each cube, etc. Thanks.

Forum Surfer
Forum Surfer

I believe unmanged switches are ok. Small being less than 20 and not connected to a larger network in any way. Now if you have a larger network, I still won't use unmanaged switches even if this particular segment only has a handful of drops. In these situations both cisco/linksys, nortel, netgear or whoever all make small, affordable managed switches. On some networks I've been thrown into in the past I've seen one unmanaged switch go bad and cause spanning tree to rebuild constantly if proper precautions weren't used by someone who deployed configs before me....which made me chase my tail and cause a $30 solution to bring a large network to it's knees. For less than $100 you can still have a managed solution even on a small segment or network.

Eric.Quam
Eric.Quam

We experienced this as well and I have never fully understood this problem. Why is the Home Router's DHCP suppressed for local clients, while the Corporate IT Router is suppressed for clients located on the larger Corporate LAN? This is counter intuitive to me. The problem would have been easy to track down if any of the computers physically located in the same (rather large) lab would have been affected. Instead the affected computers were physically located in seemingly random locations through out the rest of the building.

Forum Surfer
Forum Surfer

We constantly have to review vendor bid specs company wide. There have been numerous events where the vendor wants to throw in unmanaged switches and put them on our network. We have a policy in effect that covers this, but vendors still try to slip that in. We even had one go before the board and try to argue that their unmanaged linksys switch was every bit as capable as our more pricey managed solutions.

seanpmassey
seanpmassey

The main reason this lesson has to be relearned is because companies only look at two things on projects - cost and payback. If its working, and it isn't some regulatory requirement, you have to fight to get money to get the proper cabling and equipment.

bandman
bandman

Despite the fact that I'm familiar with everything you just wrote, I was completely ignoring it on my network at one branch. I've been having problems with my windows users obtaining DHCP leases on boot. release and renew worked fine, and I've been troubleshooting the dhcp server because I'm dumb. Thanks a lot, I owe you a beer.

bmaryott
bmaryott

Had a work area with several ethernet cords plugged into the wall, waiting to be used, as well as several spare jacks. One day a grandmotherly type decided to cords just lying there into the unused jacks "to make it look more tidy". This was an unmanaged network I had inherited, so the results were predictable. ARGH!

yogi_john
yogi_john

I learned a lot from all the comments and this is a timely article for me. We are currently looking at upgrading to gigabit ethernet (from 10/100). My switch choice originally included unmanaged, SMART managed and full managed. I had already crossed off the unmanaged switch, even though it was 1/2 to 1/10 the cost. I'm glad to know that was the right choice. Perhaps you can help me learn even more (or direct me to an appropriate newsgroup or website). We are a small company and will have about 16 computers on the LAN. Most traffic will be to the file server and to the network printer/scanner. Most of the devices are within a 150-foot cable run of our broadband router, which is also our current central switch with the nearby devices hooked up to it. The devices in the 100 to 150 foot run are connected to hubs or switches. My current thought is to buy a 24-port gigabit switch and place it near the broadband router. We would run some additional cable so all devices can connect directly to that switch. There are two distant computers we might add: one would be about a 250-foot run into a unit in the same building but touching one of our other units only by the corner (think adjacent black squares on a chess board). The other is probably over the 100-m limit for UTP and is in a separate building. We have a cable run under the driveway between the buildings. My questions: 1. Pros and cons of SMART managed versus full managed switches. 2. Pros and cons of the single switch setup versus a switch near one cluster of devices (including the broadband router) and one near the other cluster (about 80 feet between these two switches--this second switch would be closer to the two remote computers). 3. Best way to handle the two remote computers. It sounds as though running fiber to the other building is best. Thanks for your help.

Forum Surfer
Forum Surfer

My organization once took over a network that was technically not part of our company. They were such a key part of our business it was decided we would handle their networking tasks, secure their network and provide them with a link back to our side. What a nightmare. Not many drops, maybe 200. But other than a router provided by the isp, not a single managed device in sight. At least we got to start from scratch! Come to find out, the pc guy solved most problems by power cycling or having the users power cycling switches in their areas. Hard habit to break come to find out. I had one manager that would get a key from maintenance and power cycle a 3560 every time he thought his "internet was a little slow." The room was secured, but not stubborn manager proof. It took a chance run in with the maintenance guy to figure out why this device kept rebooting.

CG IT
CG IT

Spanning Tree Protocol [STP] is used to combat looping. Consumer level switches and routers do not have STP available.

s3ns4i
s3ns4i

I just started working for this company about a few months ago. It's a small business with about 100-150 nodes and the entire network is hooked up through unmanaged switches. Well today someone causes a broadcast storm on our network and let me tell you that was a pain in the ass. I literally had to go around to every damn switch in the place and reboot it to get it back on the network and a few hours later I was STILL having packet loss on the network. After the situation today I was given the funds to go and buy some managed switches. Needless to say who caused the storm? My boss! We have about 15-20 unmanaged switches located on our network :(

Deadly Ernest
Deadly Ernest

placed on the end of the drop, and use it to create a new segment for that unit. One place I did some work at, I used a home ADSL modem/router they had lying around to provide better control of the unmanaged switch they had in the office on the third floor. Plugged the drop into the router, set it up to be the local DHCP server, plugged the switch into the router. The other end was into the main router on the fourth floor. The company had expanded into offices on the floor below, taking up about a third of it. I set up each floor as a separate segment with only the two routers talking between floors - 172.1.4.x and 172.1.3.x

seanferd
seanferd

Those computers were the first to renew?

pgit
pgit

When I used to have to go before a board of directors I was told by my boss to throw all the technical facts my conclusions were based on at them. The more jargon packed, the better. Then, of course, you get to the bottom line, the recommendation. My boss (a board member himself) said these big shots don't want the others to know they had little clue what I was on about. I was also told to double up on the technical aspects if questioned, and he coached me on delivery; look and act as though the questioner understood everything perfectly. The fellow will usually act as if he does, in order to vie and jockey with the others they way those folks do. Sadly, it's not the technical details, but whether they LIKE YOU at that point that determines the outcome. In your case, it would boil down to a simple matter of whether they liked this vendor rep or not. Another factor, though, is that boards, CEOs and such tend to want to believe anything that runs counter to what their own paid staff tells them. They are "options" thinkers, if your department is unified in promoting concept A, the first guy to come along and advocate B will get an ear from these people. They heard "blah-blah-blah- equally as capable as your Cisco routers at 1/10th the cost." Minding the 'rule' above if someone asks "really?" a good rep will double up on the argon ending with "..so, yes, it has the same throughput and average uptime is an improvement, actually..." or similar. If you're familiar with the old "Baby Huey" cartoons, that's a good image to hold when dealing with boards of directors and the like.

cmiller5400
cmiller5400

Gee I bet that was an interesting board meeting. Not too many executives want to hear about the differences between unmanaged and managed switches and the lots of problems that they cause.

sidekick
sidekick

I inherited a network where 2 buildings were joined by regular CAT5 through underground conduits. My meter reads the length of one segment to be 380 feet. There is also a switch in a secure room somewhere (still don't know where) that connects this to another segment going to the server room. I don't have access to the room because it also contains equipment related to building security. So now, if it rains too much, causing the connection to go down, I need to find the maintenance/security guy to go check the switch and give it a power cycle, when he gets a minute. Maybe if they bothered to spend the money on fiber up front instead of buying a spool of CAT5 and having maintenance run it....

b4real
b4real

Always seem to be some variant of the standard - and that is what kills the network. Further it drives me crazy - and it sounds like it does for you as well. But you are not dubm!

b4real
b4real

In the rare event where someone uplinks two unmanaged switches -> the traffic flood can occur but only would effect the nodes on that switch.

Eric.Quam
Eric.Quam

Hi seanferd, Yes that does explain the pattern of failures throughout the rest of the building. My question is specific to comparing the different DHCP behavior inside the lab vs. outside the lab. (I am over simplifying the situation for clarity.) DHCP requests from inside the lab continued to behave correctly. Thanks

pgit
pgit

And that savvy vendor knows that if your BS meter goes off you are NOT going to call him/her on it in front of the higher ups, because it will just be seen as a pissing contest... and again the brass always suspects those on payroll FIRST, as in "new blood" might just stir things up and bring improvement. (for it's own sake usually) I don't baffle 'em with BS, I'm just saying a lot of vendors know the nature of those "a-gamers," and office politics and use that as much as the quality of a product to land a contract.

Forum Surfer
Forum Surfer

Mainly because this isn't the first time I've faced a similar situation. I had one incident where the vendor actually made up his own facts on the spot. Now if you get into an argument in front of the directors with a liar, even if you are right they stop listening to either of you and view it as a pi$$ing contest. I try to keep it simple and stick to the facts. I explain everything on a non technical level they can understand such as manageability, reporting to aide in trouble shooting, up time and first and foremost security. Since we abide by HIPAA standards, security is my ace in the hole. Not to mention the fact that our bourd is comprised of some highly intelligent individuals. Mind you, they have the technical ability of my 5 year old...but they bring their a-game all the same. They know when someone is throwing a bunch of tech facts at them just to try and confuse them and they know bs when they hear it.

Forum Surfer
Forum Surfer

Of course they brought up the fact that we were arguing the merits of a $50 Linksys switch versus a $5000+ Cisco switch made by the same company. I won in the end...but the price point is a big one. If you're dealing with management that doesn't see the importance of security, scalability and reliability I can see where some end up with a building full of unmanaged junk.

bandman
bandman

Thanks, I'm happy to help :-) With unmanaged switches, it's unavoidable. So you should never do it. With managed switches, as the commentators above me said, you can get away with it by using STP, but not all managed switches can do that. If the switches don't support it, then you can't do it. STP is the solution to the problem of switching loops. I hinted at the bottom of my post, but it is possible to double bandwidth between switches if your switches (both) support link aggregation. Link aggregation is basically the same thing as bonding network cards in a server. You are saying "these two ports need to act like one network port". The consequences are that the MAC table is identical, and the load is spread across both ports (as long as they're both functioning correctly. If one goes down, the load halves, but you don't lose your connection). Does that answer your question? Incidentally, if you like learning about stuff like this, I have a blog ( http://www.standalone-sysadmin.com ) that you might find interesting.

sidekick
sidekick

As bandman said, you can aggregate a couple of ports (also known as port teaming). Basically, you configure the switch to use two ports as though they were one. This would need to be done on both switches. I've never done it myself, so I can't speak to how effective it is.

b4real
b4real

Managed switches can prevent this by STP and portfast.

erilaurence
erilaurence

that was awesome awesome explanation, just some questions though, is this true for both managed and unmanaged switches? and is the reason why this happens (people adding extra links) is because some people think that this will increase bandwidth? is it even possible to double bandwidth between switches?

bandman
bandman

To know why this is a bad idea, you've got to know how switches work on the back end. A switch maintains a table in its memory of each port, and of all the MAC addresses that it knows about for each port. This way, when it receives a packet destined for MAC address A, it looks up that MAC in the table, and says "oh, MAC A is on port 12" and it forwards the packet to port 12. Now, imagine you have two switches with one cable between them. The cable between the two is connected to port 12 on both switches. So switch A's table has all of the computers connected to switch A, with the ports that they're plugged into. It also has all of the MACs of the computers on switch B, and they're assigned to port 12, since, from the switch's perspective, they're all sending and receiving traffic on port 12. This works fine, but when you add another link between the two switches, things go haywire. Now, switchA can see all of its computers, and it's got all of switchB's computers in port 12 AND port 11. What is worse is that we've got a loop. Whenever something on switchA sends a broadcast, it goes from switchA to switchB via ports 11 and 12. On switchB, the broadcast that came in on port 11 goes out on all of the other ports, including port 12, and vice versa. So of course, switchA receives the broadcast from switchB on port 12 and broadcasts it on all of its ports, including port 11. You can probably see where this is going. Spanning Tree Protocol (STP) manages this by monitoring links and figuring out where loops happen. If switchA and switchB had STP enabled, the switches would figure out that port 11 and port 12 cause a loop, so one of those ports would get shut down. For the purposes of this exercise, we'll say port 12 gets shut down. All of the traffic will then transfer between the two switches through port 11. If something happens to the cable connecting port 11, the switch remembers that port 12 was also a path, so it turns it back on. It's a good way to build redundant network connections, but lots of people accidentally incur broadcast storms by trying to double their bandwidth between switches using two cables. To do that, you've got to aggregate the ports, if your switch supports that.

erilaurence
erilaurence

how can two unmanaged switch be uplinked? is it the one described by sidekick wherein you plug back in the other end of the cable to the same switch? is this how looping occurs? or is there another scenario?

Editor's Picks