Broadband optimize

Siege warfare IT project: A cautionary tale

Consultant Bob Eisenhardt shares how a client's simple request for a public Wi-Fi turned into a nine-month nightmare project.

My wife and I arrived home in January of this year after a 10-day trip around Israel with our rabbi as our group guide and a 12-hour plane flight. I was still tired the next day when the phone rang with what turned out to be a nine-month project during which I felt akin to Buster Keaton racing through a hurricane. Gentle reader, I am sharing my story of this siege warfare IT project in the hopes that you'll learn from my experience.

Stage one

My largest client had a rep from their bad ISP provider stop by two weeks earlier than scheduled and install a new modem-router. The client called me because they had some Internet issues.

While still on Masada time, I drove up and made the appropriate modifications. The device was a Westell 7500, a combination modem-router with a public and private LAN but only a private Wi-Fi. I changed the Wi-Fi to match internal settings and set up port forwardings and departed. Like Peter Falk as Columbo, they asked "Just one more thing, a small matter. Can we have a public Wi-Fi for our waiting room?" My answer, "Of course, it will be easy."

It would be easy if the client had my good ISP provider connect and a cable modem Arris TM802 with a lovely Netgear WNDR3400 wireless router, which easily provides for private and isolated public Wi-Fi with real throw power. It took eight months to get there.

Stage two

The client purchased a similar router (that was later sold out to another client), and I sat down to bridge the Westell to the Netgear. I could get the wireless network to be 95% connected but never quite through to the Internet. I tried DNS settings, changing addresses, and numerous albeit frustrating modifications, none of which worked.

The bad ISP support tech would only talk about their Westell and go no farther. The Windows server was providing DHCP and DNS addressing. It had to be an isolated public network so that HIPAA data on the server could not be accessed. After 10 hours I threw in the towel.

Stage three

I set the project aside, because new ideas often surface in a relaxed atmosphere, but none did. Later, the bad ISP techs told me the Westell 7500 was incompatible with Netgear routers.

As the Westell used a simple RJ11 phone connect, Speedtest showed downloads of 4 or 5 mpbs. I advised the client to consider a new ISP provider, but the client stayed with the bad provider and installed a new office phone system the staff absolutely hated.

Stage four

A new modem-router arrived onsite (unscheduled as usual), and I was once again called to duty. The bad ISP tech told me I could not have access to the router and that the router had to control DHCP addressing. I refused the upgrade, advising the tech that I would never surrender those control functions at 15 years onsite. The bad ISP tech then said "well, we shall see about that." The conversation turned hostile and after a heated argument with the owner and the bad tech, the new device was set aside. To my delight, the departing tech then made an ad-hoc modification that collapsed the client's entire phone system for a half hour. Some things do indeed happen at the right time.

Stage five

At my urging, the client made arrangements to throw out the bad ISP service, and a date was made to install a proper cable modem-router that could provide wireless, public, and private from the good provider I have at home.

The new device arrived onsite (unscheduled again), and I waited two hours for a delayed tech. The tech installed the device, provided me with all the IP addresses for the new network, and left. All of the IP addresses were wrong! Three tech support phone calls the next day were worthless until I spoke with a tech who gave me everything I needed to know and had fast Internet directly through a laptop. I documented those numbers and saved them.

An associate owner said he had a sealed Belkin router I could use (they did not want to purchase if they did not have to, which was just penny pinching). I was not surprised that the smaller router was a home-toy device and did not do public and private Wi-Fi. I took the Belkin router for my home network and brought my Netgear WNDR3400 up to the client site, connected it to the good ISP provider modem router with five dedicated IP solid addresses, and it worked fine. The Speedtest was upwards of 50 gpbs download.

I drove up on the morning of Yom Kippur to complete the task and discovered I had left everything at home, as fasting is a huge disruption of my morning routine. I decided that now God was against my working this day and wisely went home.

Stage six

A few nights later I drove up to connect up the Netgear to the laptop for one last test, and it was perfect. I connected the Netgear to a gigabit switch and deleted the bad ISP provider DNS servers in forwarding, replaced them with two good ones, and finally snapped off the cursed Westell 7500 for good. Without any further modifications, the Internet flew at every workstation, and wireless Internet was even faster than their previously wired connect. Public and private Wi-Fi was easily enabled just as I wanted it to be eight months ago.

Summary

I attempted many things on this project; it was a delicate balancing act of listening to the client, working with multiple tech support idiots, carefully analyzing brick walls, angering the Gods, and having a satisfactory payoff visit in the last two hours of the project. And it all started with a simple request for a public Wi-Fi after a trip down from Masada.

7 comments
JohnMcGrew
JohnMcGrew

...and the ISP that insists that it be used. It was quite the unpleasant surprise the first time I discovered that you can't turn off DHCP, which means forget using DNS and a Windoze domain on that subnet. My usual solution for small clients that are stuck with this ISP (for whatever reason) is to create a proper network behind a separate router of my choosing. The Westell network can be "public" and the inner network is secured. It's a pain to set up routing through the Westell, but at least it works and you are reasonably isolated from that ISP's forced insanity.

viProCon
viProCon

First, you mentioned near the end of Stage Five, "The Speedtest was upwards of 50 gpbs download". 50Gbps WAN link?? Please clarify if this is correct because if so, I am moving to your city. Second, how did the client react to you through all this? Often clients are ignorant of who is at fault, they see you onsite working the issue and say "hey, why aren't YOU getting it done?". Did you have to deal with that much?

bkindle
bkindle

I am in an almost identical situation right now, except I am at the end of stage 1, maybe 2. This gives me hope but gives me heartburn too knowing that I have to go through the same hoops and walls :(

robo_dev
robo_dev

You create a good design, test it, and put it into production. Granted I've been doing this since Bill Gates was a kid, but seriously, muddling through it is the wrong approach. Reading this case study I see two issues: 1) if you don't start with a plan with solid requirements, you can really waste time trying to make a Toyota Corolla ready to compete in the Indy 500. 2) Often businesses and business owners have really no clue about technology. Assuming you can run your business on a $20 router with a 50-cent power supply is naive, at best. You get what you pay for. If, at day one, you said, look, here's a Cisco router and a Cisco WLAN system. It will cost you $1500, but when I come back in a year, it will still be running, and it can be configured or re-configured to do just about anything. If you cannot afford it, here is a lower-cost Sonicwall, Aerohive, or Aruba solution. Trying to use consumer-grade equipment for a business is a huge mistake. Most of it does not have the feature-set you need, and most does not have the stability and reliability/redundancy features that are also needed. Been there, done that. To race in the Indy 500 you start with a race car and refine it, don't try to muddle through to try to make your Honda Civic look like a Honda Indy car :)