Network Performance problems

The Resolver has worked on this problem without satisfactory conclusion. It is now the aim of the PM to attempt alternative areas for research into this problem. More information can and will be provided to this issue when requested. Unfortunatley that could not be documented here as a result of the 2000 word limit.

2.1 Mis-configured Firewall / Proxy server
Support has worked closely with the firewall / proxy server manufacturer, WAN Support and the firewall UK reseller to identify hardware problems or mis-configured components within the firewall itself. To that end funds were spent in order to produce a firewall Health Check.

Additionally BorderWare support in Canada has been asked to comment on the firewall configuration and have confirmed correct configuration of this device. An additional patch has been installed on the firewall to exclude www Internet users from accessing the external DNS service homed on the external interface of the firewall. The patch allows an ACL to be applied, thereby allowing internal customers access to the external DNS whilst others on the www side are prevented from doing so. This was required by the ISP to reduce a perceived Denial of Service problem that was the result of the firewall having good bandwidth access and a well functioning DNS forwarder accessible by www users. As the patch resolves the DNS access issue and the problem presented here is purely IP in origin, this would not seem to be the cause of the current problem. Additionally the problem has existed for at least a year, prior to any significant patching of the firewall in January 2004.

A BorderWare engineer has also visited (Feb 2004) to look into the firewall / proxy server too and has also confirmed correct configuration.

2.2 Mis-configured Comms equipment within the Wide Area Network
Previously it was noticed that moving files between a Core switch and the Firewall proved problematic. This was identified after some investigation to be a mis-configured network port on the switch set to 100Mb/s Half Duplex, when the value should have been 100Mb/s Full Duplex. No other mis-configured devices have been reported, identified or relayed back to our support teams.

The issue with the above report was that this mis-configuration was only identified after it was reported. It was not obviously faulty and would have caused a problem for a custoemr site plugged into that port. It is possible that other issues like this may exist and have not been discovered through the investigations currently taking place.

When the problem was re-raised in January, the resolver discussed the problem with the switch manufacturer Alcatel. The configuration of switches was changed to include the MAC address of the firewall / proxy server so that discovery by hosts was not required. The inclusion of this information seemed to make the problem worse in that previously the firewall responded to ping requests eventually. Once the configuration was changed only the gateway switch could be pinged in order that a connection to the firewall / proxy server could be made.

2.3 Mis-configured Comms equipment within the customers Local Area Network

One customer site was used as a test bed by the Resolver Team. All Comms hardware within the sites Local Area Network was replaced:
q New Alcatel switch
q All unmanaged hubs and switches on the curriculum LAN were replaced with managed ones
q All hosts were re-configured from auto-negotiating their network speeds to these set to 10Mb/s Full Duplex

The test seemed to provide positive results in the short-term. However over long-term use the problem as described above re-occurred after a few days, thereby indicating that the LANs within customer sites cannot be the issue here. Additionally no test results have been submitted that show the breakdown of testing components individually to see if correlation exists between replaced hardware and the problem itself

2.4 Systems Integration issue between components on the WAN; Firewall / Proxy and Alcatel core switches
The BorderWare engineer has also visited to look into the firewall / proxy server and provide onsite technical assistance to the Resolver Teams. This meeting of technology specialists also included Alcatel so that the systems integration issue between hardware manufacturers could be examined. A brief summary of actions was submitted by Astro for this and these communications are detailed below.

From a mail submitted by the Resolver on 27th February 2004:
q As you are aware on Wednesday the Borderware engineer and the Alcatel support engineer both attended to help with the investigation into the customers Internet ongoing problem.
q Prior to their arrival all eqpt stats were logged and all core switches were reset to allow all parties to monitor and view errors as they occurred.
q Both parties set up test eqpt and logs to try to capture the problem when it occurred.
q Unfortunately the customers were informed there would be network outages during Wednesday and most decided not to use the network.
q This created a situation where the network was only very lightly used during the day of testing and the problem did not occur.
q Some batch scripts were left running on Borderware giving stats to the support team when the problem re - occurs.
q The Alcatel engineer went through all configurations and couldn't find
any problems with the configuration of the network.
q He left some stats collections running to assist with fault isolation when it re - occurs.
q Unfortunately to date this fault has not re - occurred, but feel we have the necessary tools in place to capture it and the support of the 2 manufacturers to help us resolve it .

Following a separate conversation with the BorderWare engineer the following issues were raised by the Project Manager on 1st March 2004:
q The core switches were rebooted because they were not working. Either 1 or sever

or several were not responding when the engineer visited. This was shown to be the case as no packets whatsoever were reaching the BorderWare firewall

q Usually a command is issued to the switches to reboot them. As the BorderWare engineer understands following an inability for these to be rebooted using command line, the devices were physically rebooted.
q The Alcatel engineer informed the Resolver that unless the switches were configured in the same way, then spurious errors will occur, the nature of which he could not define. To that end the core switches were rebuilt

The Resolver responded with the following clarification on the 1st March 2004:
q When I stated the core switches were re?booted to reset the statistics.
q This action took place for the reason stated actioned by our engineer at approx 7am, which was 3 hours before the Borderware technical support engineer turned up on site.
q At that time our engineer inserted an Alcatel 4024 edge switch in line between the Alcatel core OmniSwitch and the Borderware to allow both parties to have a testing position to plug their network analysers and test tools etc, to assist in fault localisation.
q When the Borderware engineer was on site the temporary testing switch locked up completely, this was cold re ?booted -(not the core switch), but when it didn?t come up immediately, it was decided to remove this switch from the equation, bearing in mind the Borderware engineer was not intending to use any external networking tools or analysers.
q Point 3 ? The Alcatel engineer did indeed state that the 802.1q groups needed to be created in the same order at both ends of each core, otherwise spurious errors could occur. (An un-documented feature)
q As we could not guarantee these were created in the same order and had been added to over the years that the network had been in existence, it was decided to re ? create these groups in accordance with Alcatels recommendations and as a matter of best practice.

q The above issue relating to 802.1q grouping would only affect the three remote core sites and as you have stated, that the problems are occurring on sites coming out of all 4 core sites.
q Also this would affect the admin networks on all sites and also the corporate users that share these links, although hopefully this will improve matters, we feel we have not got to the cause of the problem.

Every indication from the Borderware engineer would suggest that the firewall is correctly configured to ?integrate? with the Alcatel switches used in the WAN design. Additional reports are now produced by the firewall when a problem occurs at this device. However this will not report any trace if the problem occurs elsewhere. To date no major fault traces have been captured by the firewall in light of this problem.

2.5 Systems Integration issue between components on the WAN; Alcatel core switches
Please refer to the issues above under Systems Integration issue between components on the WAN; Firewall / Proxy and Alcatel core switches. The Alcatel engineer did indeed state that the 802.1q groups needed to be created in the same order at both ends of each core, otherwise spurious errors could occur. (An un-documented feature). The question here is how many other undocumented anomalies may exist that could contribute to the current problem. The core switches configurations have been rebuilt so as to counter this particular unknown feature.

2.6 A host operating system software issue
No specific Operating System issues have been established by ourselves or by the 3rd party Resolver. The Resolver has indicated that a problem with the Windows 9x architecture has been recorded as a problem when connecting to the Internet. This was given as a source of investigation by another network specialist workign for the Resolver. To date no documented evidence from the Resolver of this issue has been submitted to ourselves.

2.7 A problem associated with the BT links between aggregation points and gateway Internet connection
No reported BT Link issues have been reported as part of this problem by the Resolver. Cold weather has recently been the cause of network outages around the borough however. It is not understood how environmental effects could result in an outage of this type. The expectation is that the use of fibre for these links significantly reduces the problems associated with other connectivity media such as radio or satellite links which have been known to cause the problem seen above in other networking environments.

My gut feeling is that it is an ARP issue. I would look at the arp databases on all of the workstations, switches, and routers and try to determine which one is having problems.

I like your idea although we suggested a routing issue to our Resolver. It cannot be at the workstation level because we can initiate a connection for all hosts at a site by pinging from a single workstation or server. The incosistent nature of the fault is weird too. How could we replicate an arp problem in the way you are suggesting?

Hi I'm a network Engineer at a major telco in Europe and before I start to troubleshoot I would like to have the following :

A Conceptual diagram of your network please remove public IP's from this diagram but keep internal ones and masks if it is allowed by your security policy, second a detailed inventory of your hardware / type of WAN and internet access

Some packet sniffer dumps would be a must most revelevant if captured between the Internet Router and the Firewall and between the Core Switch and the Firewall. This dumps must be taken in the time frame when the problem is happening.

If you allow me I leave a comment :
Any credible SLA states a maximum of 24 Hours to solve a problem your suppliers are taking half an year ... Nope that's not the way better start thinking in getting new suppliers.

Send me a e-mail if you think my requests are within your security police to f.jose.oliveira@ptprime.pt

I would start with a tracert from a client to the proxy/firewall to see exactly where the your packets are getting lost.

I would start with a tracert from a client to the proxy/firewall to see exactly where the your packets are getting lost.

FIRST YOU HAVE TO CHECK UR CAT-5 CABLE WHICH U R USING TO CONNECT TO THE LAN. IF IT IS ALL RIGHT THAN U HAVE TO CHECK THE GATEWAY SETTINGS IN THERE IF U DONT PUT THE PROXY ADD. YOUR HOST WILL NOT ABLE TO CONNECT WITH THE INTERNET. IF IT IS ALLRIGHT THAN YOU HAVE TO CHECK YOUR PROXY SERVER IT MIGHT BE NOT INSTALLED PROPERLY.

Why not move a PROXY server to each remote site?

That way the workstation and the proxy will both be local to one another. (ON THE LAN side)

If original problem is timeout reaching proxy resulting in NO SURF then why not move the proxy server around in the network till you get a satisfactory result? (move it closer to workstation) (eliminate any core switches, or large switches)

Investigate whether spanning tree could be causing your trouble. You did say you have a campus network. Spanning tree does learn, and is capable of learning incorrect information then by rebooting the ethernet switches(or ports) the problem goes away?

Interesting that a simple PING provides some interesting traffic which "NORMALIZES" the issue again.

Where do the tracerts stop when the trouble is present?
From workstation facing proxy (do tracert)

from proxy facing workstation (do tracert)

from proxy facing WAN (whatever device) (do tracrt)

all at the same time. See what you get before you reset anything. Document your results.

See then when the problem improves.