Since TCP/IP is quickly becoming the protocol of choice on today’s networks (both from a LAN and a WAN perspective), knowing how to troubleshoot problems is more important than ever. Although there are many outstanding text-based command-line utilities, these tools don’t always do the job. In this Daily Drill Down, I’ll discuss some troubleshooting tools that deserve a place in your bag of tricks. These tools aren’t the only ones you should use, but you definitely should acquire them, especially if you’re using only the text-based utilities that come with Windows 9x and NT (like Ping and Tracert). Just as with the desktop operating systems we all use, tools change over time and new ones are periodically released.
To call NeoWorx’s NeoTrace a GUI version of Ping or Tracert is to do the utility an injustice. After downloading and installing NeoTrace, you’ll quickly see how useful it is. Within seconds of entering a site name, you have a choice of viewing options. You can view:
- Each router crossed on the way to the site you’re checking.
- The IP address.
- The ISP whose network the site is running on.
- The time it took to reach each router on the way to the final destination.
NeoTrace allows you to look at as many sites as you want. This feature is useful if your company has several critical sites from which it must have timely response. The information is updated periodically, and you can quickly see whether there’s a problem router on the path to the host you’re trying to reach.
By default, the path to the host you’re pinging is checked every five seconds, with a timeout value of three seconds. After three seconds, NeoTrace assumes the node is not responding to the ping. For sites you check on a periodic basis, you may want to consider specifying a higher timeout value (for instance, 60 seconds).
You also have the option of pinging every router on the path to the host you’re watching or just the end host. I recommend starting with pinging the route instead of the host. If you’re having problems accessing a particular host site, you may find that the problem is an overloaded router and not the host site at all. You have the ability of saving, printing, or copying the route to the Clipboard. If you determine that your connection problems aren’t on your end, you can relay that information to the company hosting the site.
This tool is priced at $29.95 on a per-user basis. You can download a slightly older version of the program to see whether it meets your needs before purchasing it. I use NeoTrace on a regular basis, and to me it’s well worth the price.
This utility should be called the Swiss Army Knife of network troubleshooting tools. There isn’t much NetScanTools can’t do. There are two flavors: NetScanTools 4.0 and NetScanTools Pro 2000. NetScanTools 4.0 is helpful for those on a budget, but once you’ve taken a look at the Pro version, you’ll want to spend the extra cash. NetScanTools 4.0 sells for $25, and you can download it from NetScanTools . NetScanTools Pro, at $150, is available on CD-ROM only. For an additional $30, the Pro version includes a software maintenance program. With this program, you’ll receive a CD twice a year that contains new features, updates, and patches. For the purposes of this Daily Drill Down, I’ll concentrate on NetScanTools Pro.
The usefulness of this tool is limited only by your imagination. Each of the tabs has two buttons that will get you additional information on the task you’re currently working on: RFC and Help. The information that’s available by clicking the Request For Comment (RFC) or Help button changes based on the active tab. You won’t receive a printed manual when you purchase NetScanTools Pro; it’s online.
The Email Validate and SMTP Email Generator features come in handy when you’re debugging e-mail problems. Email Validate may or may not work, depending on how tightly the e-mail administrator has locked down the security on the e-mail server. SMTP Email Generator allows you to see what’s going on with a mail server at a low level. After you send (or attempt to send) an e-mail message to an individual, you can click the View Log File button and see a blow-by-blow record of the send process. This record shows you how the mail server responded to each inquiry you sent.
The TCP Term tab offers options for testing services on a particular port to see whether the services respond to inquiries. For example, if you need to see whether your SMTP server is responding to requests, first enter the mail server name in the Host Name or IP Address field. Then, type 25 in the Target Port Name Or Number box and click the Connect button. If you’ve entered the correct information, and the host is listening on the desired port, you should get a reply back shortly, confirming that the other end is accepting input. Once you’ve finished working with TCP Term, click the Disconnect button.
Because timing synchronization is so important, you should know how much the time on the system that runs NetScanTools Pro differs from actual time. You have the ability to send and receive time requests using TCP, UDP, or SNTP (Simple Network Time Protocol). NetScanTools offers a built-in list of approximately 58 timeservers ready for you to use. Fair warning: Not all timeservers respond on all ports, so until you become familiar with how timeservers work, you may have to do a little experimenting to find a combination of timeserver and port that works for you. On the bright side, NetScanTools Pro lets you test whether you can receive a time sync signal through your corporate firewall.
When the timeserver responds, you’ll see an item called Stratum. This item tells you how close the timeserver is to another timeserver that’s using an atomic clock as its time reference. Stratum 2 indicates that the server in question is one step away from another time clock that’s using an atomic clock for time reference.
Although NetScanTools Pro’s Port Probe can be a dangerous tool in the wrong hands, it can be invaluable if you’re charged with keeping your network secure. You can use this tool to determine what type of “holes” exist in your firewall and servers you’re responsible for keeping secure. By default, Port Probe starts at port 1 and stops at 256. Depending on what you’re testing for, you may need to change these values. Port Probe is also a good way to ensure that a specific service is running and ready to receive requests. Simply test for a specific port number on a designated host name or IP address.
The Ping and TraceRoute tabs offer alternatives that are faster than using the Ping and Tracert text utilities that ship with Windows 9x/NT. If you start out working with an IP address, NetScanTools Pro will be able to resolve the IP address to a host name (if one exists).
Name Server Lookup lets you get in under the hood on a DNS server to see whether it knows about a particular domain. If your company’s DNS is hosted by your ISP or another third party, Name Server Lookup tells you exactly what’s in your DNS record. It never hurts to do a periodic checkup on the primary and secondary DNS servers to ensure that the information being given out is correct.
If you think you may be having problems with your DHCP server giving out addresses, you have a couple of options—and you don’t have to reboot several workstations or work with the DNS Server configuration. By using the options on the DHCP tab, you can verify that the DHCP servers on your network are actually responding. Simply specify an IP address range, and NetScanner will report which addresses are active and, where possible, resolve the names from the IP address.
I’ve barely scratched the surface of what NetScanTools Pro is capable of doing. One of the best things about this tool is that it comes on one CD, which you can slip in your pocket or a thin briefcase and have ready to use at a moment’s notice.
Sniffing your network
Occasionally when troubleshooting, you may need to see the traffic on the wire in order to identify the problem and fix it. This is where a piece of equipment known as a protocol analyzer comes into the picture. There are two kinds of analyzers: hardware based and software based. The software-based protocol analyzers are less costly than their hardware-based counterparts. A software-based analyzer will run between $1,000 and $5,000 on average. Hardware-based analyzers range from $12,000 to around $20,000, depending on the type of maintenance agreement you select. (A maintenance agreement generally covers new or improved protocol decodes in addition to program fixes or updates.)
Quite a few protocol analysis tools are available. You can view a good representative list at NetAnalysis. For those working with a mostly Novell network, Novell’s Lanalyzer for Windows provides a good introduction to protocol analysis. When you start moving into a multiplatform network, you may find that you’ll have to move up to a more advanced analyzer, such as Network Associates’ Sniffer Pro .
When most people hear the words “protocol analyzer,” they automatically assume you’ll be looking at bit-level information in the packets to find the problem. When I’ve used analyzers in the past, I‘ve found that the best approach is to watch the packets that are captured and learn what’s “normal” for your network. For example, a couple of years ago a bank called me in for troubleshooting. Every PC in the bank was locking up about every 15 minutes and had to be turned off and back on. The bank’s system administrator assured me it was a network problem because he’d checked all the network and electrical wiring. Needless to say, this qualified as an emergency situation from the bank’s perspective.
It took me only a few minutes to get my analyzer up and running. However, the file server wasn’t showing any utilization that I considered out of the ordinary. I started up the analyzer in Packet Capture mode and left it running while the administrator took me on a tour of the building. When we returned, I stopped the packet-capturing process and examined the traffic that had been on the network. After a few minutes, I noticed an odd pattern. I observed that several workstations were requesting the current date and time from the file server between 20 to 30 times per minute. I repeated the packet-capturing process, but this time, I filtered the traffic captured. I wanted information on the workstations identified in the initial capture and the file server they were talking to.
The second capture came back the same as the first. Requesting the current date and time is normal when a workstation first logs into the network. However, these repeated requests were unusual. The administrator told me he was running an application that enables the workstations to get the current time between two to three times per day. The purpose was to ensure that financial transactions were being posted at the correct time. I suggested we disable that service on one or two workstations. Amazingly, the workstations that had been locking up at 15-minute intervals ran over an hour without a problem when we stopped watching them. At that point, I recommended that the service be disabled on the remaining workstations until the application vendor could be contacted for assistance. Needless to say, I got a call from the bank a few days later requesting a copy of the analyzer I‘d used so the administrator could handle this type of problem next time.
Not all problems will be on your local network. In such cases, you’ll need WAN functionality from your analyzer. You should be able to look at traffic passing on a WAN connection or on a specialized connection, such as Gigabit Ethernet.
Dealing with TCP/IP problems can quickly become confusing and overwhelming. However, there are tools you can use to make the task easier. Don’t expect to become an expert overnight in the field of protocol analysis. I’ve been using protocol analyzers for over five years, and I’m still learning.
Ronald Nutter is a senior systems engineer in Lexington, KY. He’s an MCSE, a Novell Master CNE, and a Compaq ASE. Ron has worked with networks ranging in size from single servers to multiserver/multi-OS setups, including NetWare, Windows NT, AS/400, 3090, and UNIX. He’s also the help desk editor for Network World. If you’d like to contact Ron, send him an e-mail. (Because of the large volume of e-mail that he receives, it’s impossible for him to respond to every message. However, he does read them all.)
The authors and editors have taken care in preparation of the content contained herein, but make no expressed or implied warranty of any kind and assume no responsibility for errors or omissions. No liability is assumed for any damages. Always have a verified backup before making any changes.