You finally took the plunge, wrote the big check, endured the chaos and confusion of dealing with the phone company, ISP, and perhaps a handful of third-party contractors, exercised your patience, and now your T-1 line is installed. It’s fast, it’s reliable, and now your network can live happily ever after. Well, maybe. Despite its high marks for stability and reliability, T-1—like any other technology—is not invulnerable to problems. Although you’re likely to have far better uptime with T-1 than with lower-cost connections such as analog lines, DSL, and cable, your connection can still go down. What do you do when that happens?
In this Daily Drill Down, I’ll discuss some of the problems you may encounter with a T-1 connection, how to determine where the problem lies, and what you can do about it.
Whose problem is it, anyway?
The first step in troubleshooting any Internet connection is to determine whether the problem is at your end or elsewhere. When your T-1 connection drops, the problem can be at one of several locations:
- The telephone company central office (CO)
- The local loop (the line from the CO to your site)
- The demark device, a “smart jack” or other equipment installed by the phone company, either on the outside of your building or in your phone/server room or closet, that makes the line of demarcation between the part of the line that is the phone company’s responsibility and the part that is yours
- The customer premise equipment (CPE) (your CSU/DSU, router, and other equipment on your premises that connect your local area network to the T-1 line
- The Internet service provider (ISP) that provides the connection to an Internet backbone network
Analyzing points of failure
It’s important to determine the point of failure first; otherwise, you may:
- Spend valuable time trying to “fix” the problem at your end when you have no problem at your end (perhaps creating a new problem with your “fixes”).
- Call out the telephone company or your ISP when the problem is actually with your own on-site equipment or configuration.
|A normal TRACERT to www.yahoo.com|
You can often use basic TCP/IP tools to help determine where the problem lies. For example, when I recently lost Web connectivity on my network, I first brought up a command prompt and determined whether I could ping outside the LAN. A couple of experiments showed that I got a response when pinging the ISP, but attempts to ping other Internet hosts timed out. I then tried a TRACERT to www.yahoo.com. When the connection is operating normally, this would look like Figure A.
|The trace ends at one of my ISP’s routers.|
Instead, the trace stopped at the Dallas gateway 1 (dal-gw1.august.net), one of my ISP’s routers, as shown in Figure B.
This was my signal to stop trying to fix things at my end and call my ISP. The harried tech support folks informed me that yes, they knew a router was down and yes, they were working on the problem.
If I hadn’t been able to ping my ISP, I would have known that the problem was closer to home—either with the telecom line, my customer premise equipment, or even the configuration of my computer. (In that case, my next step would have been to find out whether other computers on the LAN were also unable to get out, to eliminate a computer configuration problem.) To help narrow down the source of the problem, use standard troubleshooting procedures by pinging:
- An Internet host on the other side of your ISP (such as a common site like www.yahoo.com).
- Your ISP.
- The far side of your router (the external interface connected to the Internet).
- The near side of your router (the internal interface on the same subnet as the computers on your LAN).
- Another computer on your LAN.
- Your own IP address.
- The loopback address (127.0.0.1).
Start at the bottom: The Physical layer
The simplest problems, although not necessarily the easiest or most expensive to correct, occur at the Physical layer: The hardware is broken, disconnected, or incorrectly set up. Physical layer problems can occur on your side or on the telecom side, so it’s important to first pinpoint where the trouble is, before worrying about what it is.
Common Physical layer problems
Problems at the Physical layer can include the following:
- DTE interface cable: the cable that connects the CSU/DSU to the router, if you don’t have an integrated device
- Integrated CSU/DSU router
- Stand-alone CSU/DSU devices
- Line problems
If you have a separate CSU/DSU and router, you should keep an extra interface cable on hand to make it easy to check for a bad cable. A very common but easily overlooked problem is a simple one: The cable becomes disconnected when someone moves or works around the devices.
Your CSU/DSU, router, or integrated device will usually provide a number of alarms or lights to indicate link status and detected problems. The specifics will vary depending on your device but, for example: A red indicator might mean that the signal has been lost at the CSU/DSU (local problem), while a yellow indicator might mean the signal was lost at a location downstream on the network, and a blue indicator might indicate that the signal was lost at a location upstream of the device. Loopback testing (discussed in the next section) can be used to isolate the problem to a specific piece of equipment or a particular network segment.
Troubleshooting the telecom line can be done with a T-1 loopback plug.
Troubleshooting tools and techniques
You’ll recognize some of the troubleshooting tools and techniques used to diagnose problems with your T-1 connection as the same ones used for that purpose with other connection types. For example, a handheld network analyzer can be useful in pinpointing physical problems.
T-1 troubleshooting requires an understanding of the following:
- In-service vs. out-of-service testing: This refers to whether you must bring down the link to do the testing or can test connectivity problems while the line is up and operational. With out-of-service testing, a testing instrument takes the place of normal network traffic. The test instrument transmits known bit patterns; if the patterns deviated from what is expected, they’re considered to be errors.
- Point-to-point testing: Two testing instruments, as described above, are used in an out-of-service environment, one at each end of the link. This allows you to test the link in both directions.
- Loopback testing: Loopback testing involves having the testing instrument send a loopback code to a CSU at the other end of the link. Then data is sent and is looped back to the test instrument. It’s harder, with loopback testing, to determine whether the errors are occurring on the transmitting or receiving side of the link. Tests can be performed on the full T-1 circuit or on individual channels.
Out-of-service testing can be done either as point-to-point or loopback testing.
You can conduct loopback tests in two ways:
- Using the software loopback mode
- Using a hardware loopback device (loopback plug or cable)
Your T-1 router may allow you to use the router software to place the line in loopback mode for testing by entering a command at the router interface. An example is the Cisco IOS software. You can test parts of the circuit separately.
Loopback mode should be used only for testing purposes.
A loopback plug is a special cable with RJ-45-compatible plugs whose wires have been crossed. You can construct your own loopback plug for a T-1 CSU/DSU.
T-1 loopback plugs
To make a loopback plug for T-1 testing, strip the wires on a 5- to 6-inch long RJ-45 cable (with connector plug). Twist the following wire pairs together: wires connected to pins 1 and 4 and then wires connected to pins 2 and 5. You can also buy loopback plugs for line testing. For example, the MTM-LBT1 costs $15 to $25 and can be purchased from your network supplies vendor or online at MTMnet.
The first step in loopback testing is to test the CPE (CSU/DSU, router, and cables). If they check out okay, you may need to engage the cooperation of the telephone company to test the line itself. When you call the teleco, you may need to provide your circuit ID number. This can usually be found on a tag in the demark box where the “smart jack” or T-1 controller card is located.
You might also need to call your ISP to help with the testing if you suspect a problem at the other end of your circuit (your ISP’s router).
While in loopback mode, you can use ping with various data patterns to test the error rate of the equipment in various parts of the circuit. Detailed instructions for performing these tests using Cisco equipment are available on the Cisco Web site.
Most T-1 integrated router/CSU/DSU devices will include diagnostic routines. You can refer to your router documentation for lists of error events and alarms and what they mean.
Cisco routers use the show interface command for displaying line status information and other data that can be used in troubleshooting. For example, if the status line state shows the message Serial x Is Down, Line Protocol Is Down, this means the router doesn’t detect a carrier detect signal. Causes can range from a line that has become disconnected from the CSU/DSU to bad cables, hardware problems with the CSU/DSU, or a telecom problem.
On the other hand, if the status line displays Serial x Is Up, Line Protocol Is Down, you may have a misconfigured router, the telecom switch may be misconfigured, or you may have a timing problem.
Clocking problems and synchronization
T-1 communications are synchronous, meaning timing information is shared, so clocking and synchronization can be a problem. This is different from asynchronous communications, such as those of analog modems, which use start and stop bits included in the data stream for timing. Clocking must be properly set on your CSU/DSU to avoid errors and communications problems.
Timing problems can cause frame slips, which result in duplicated or deleted data, or even complete disruption of the framing pattern. To avoid slips, the teleco is used as the clock source, and the CSU/DSUs on both ends of the connection are configured to use loop timing. In loop timing mode, the device will use the incoming signal at the network interface to get its clocking information. If the teleco doesn’t provide a timing source, the CSU/DSU at one end of the connection can be set as master, with the other as slave. The master device is set to use internal timing, and the slave is set to use loop timing. The key is to use only one clock source on the network.
Consult your teleco to determine the correct configuration, and consult your device documentation for instructions on how to set the timing options.
T-1 connections are typically reliable and stable. When problems do occur, however, they can be difficult to troubleshoot due to the complexity of the technology and multiple possible points of failure. The first step in troubleshooting problems with a T-1 link is to determine whether the problem is with the customer premise equipment, at the demark, on the telecom side, or at the ISP.
Common problems include physical problems with the CSU/DSU, router, cables, or the line itself, and clocking and synchronization problems. You can use common troubleshooting tools and techniques, such as ping and handheld network analyzers, as well as software or hardware loopback testing, to isolate problems.