Resolving mail server problems with Nslookup and Telnet

E-mail has become a mission-critical tool in most businesses. In this Daily Drill Down, Ron Nutter shows you how to use Nslookup and Telnet to help diagnose problems when e-mail doesn't go through.

Users don’t give much thought to how e-mail works. They just type their messages, click Send, and—whoosh!—away goes their e-mail. From an administrator’s perspective, you can only hope it works that easily. Unfortunately, it rarely does.

You will face common problems, such as messages that don’t get delivered to an external e-mail address. This can manifest itself in Exchange as 550 errors, when trying to send mail. Non-Exchange e-mail servers will vary their complaints, but they will normally report host resolution errors. You may also notice e-mail piling up in the outbound queue.

DNS failures can cause these errors. In this Daily Drill Down, I’ll show you how to identify the cause of these problems using Nslookup and Telnet.

What is Nslookup?
Nslookup is a command-line text utility that translates information generated by DNS servers into a form that you can understand. Remember, DNS servers act like direction providers on the Internet—they give you a TCP/IP address for the name of the host system you’re trying to access. Because a workstation can consult more than one DNS server when looking for an address, Nslookup can help you track down where a potential problem is occurring. Nslookup can be a great help when you have problems sending mail to a particular domain due to DNS failure.

Some network administrators are intimidated by Nslookup because it’s not a GUI utility. Still others think that because it’s a text utility, it isn’t capable of doing the job. I’ve had to show more than one person that I can get the information faster with a text utility like Nslookup than they can with some of the nicer looking GUI utilities that are available. For the purposes of this Daily Drill Down, the version of Nslookup we’ll be working with comes with NT 4.0 and Windows 2000.

Nslookup has other command-line options available for other tasks. Each command-line option does something different, but they aren’t relevant to this discussion. I’m going to focus on the options that you will use on a day-to-day basis.

You’ll start Nslookup by going to a command prompt on your server, typing nslookup, and pressing [Enter]. You’ll then see what appears to be an old-fashioned DOS prompt. At the top of the screen, you’ll see a DNS server name and address. When you start Nslookup in a command window, it will look at the DNS servers that your workstation/server is currently set to use.

You can change the server that Nslookup is viewing by typing servernewdns and pressing [Enter] where newdns is the TCP/IP address or fully qualified domain name of the DNS server. When you start Nslookup don’t be overly concerned if you get a message on the screen that says it can’t resolve the name of the DNS server, yet it gives you the TCP/IP address of the DNS server in the error message. This is a feature of Nslookup, and one that won't affect what the program can do.

Using Nslookup
 When the Nslookup prompt appears, type set type = mx. You won’t get any message that your command has been accepted other than the return of the > prompt from Nslookup.

Type the domain name (i.e. you’re trying to contact and press [Enter]. Within a few seconds, you should see one or more lines on the screen listing the MX records found for a particular domain. If you don’t see an MX record, the problem is solved. Add an MX record for the target mail server to the DNS.

You can have more than one MX record listed. This can happen if a particular domain has one or more MX records. If the primary mail server fails for a domain, mail will continue to be received by the secondary or tertiary mail server. If you have more than one MX record, look closely at each line listed by Nslookup. On each line you will see a number, which might be one or two digits. This is referred to as a weighting factor for MX records. Print this information out or write it down in a note pad.

The weight factor for an MX record controls the order in which the MX records are used to send mail to a particular domain. The lowest number (usually zero) is tried first. If your mail server can’t communicate with the mail server associated with this number, it then tries the mail server with the next number. This process will continue until the source mail server finally communicates with a mail server in the list and sends the e-mail, or until the source mail server runs out of mail servers to communicate with. The amount of time the source server will wait before trying the next MX record will depend on the software your source mail server is running.

There are several reasons why a mail server might time out:
  • If the target mail server is under a heavy load at the time your mail server is trying to connect to it, the target server may be unable to respond before your mail server times out and tries the next mail server.
  • Even though an MX record is defined for a particular domain, it doesn’t necessarily mean that the server listed in the MX record is configured to accept e-mail.
  • The target server may be down and unable to receive mail.

After you’ve obtained the list of servers that are configured as mail servers in DNS with the MX record, the next step is to try to ping the mail server by DNS name. This will do two things: First, it will cause the packet sent by the ping command to go through the DNS resolution process, which will ensure that the name properly resolves to the correct TCP/IP address. Second, it will also ensure that the server you’re trying to contact actually responds.

If the ping doesn’t go through, then the server is probably down. However, it’s also possible that the server is behind a firewall that blocks ICMP traffic, which includes ping packets. In this case, the next step is to use the Telnet command.

Using Telnet to check a mail server
Normally, you’d use the Telnet command to log in to another server. In this instance, you’re going to use the Telnet command to trick the target mail server into thinking that you’re another mail server. Rather than the default Telnet port of 23, you’re going to tell the Telnet command to use port 25, the default SMTP port used by mail servers.

To do so, go to a command prompt on your server, type telnet 25, where mail is the DNS name of the mail server and is the domain where the mail server resides. Press [Enter]. Make sure you include the number 25 too, because this tells Telnet to use port 25 rather than Telnet’s default port.

You should get a response back from this attempt within a few seconds. Note how long it takes to get a response back. This delay becomes important when there are multiple MX records for a particular domain. If the server doesn’t respond, it could be down. It’s also possible that there’s a firewall blocking access to port 25, which will cause no mail to be delivered at all.

When the mail server responds, you’ll see a message along the lines of 220 mailserverversion where is the name of the domain you’re connecting to, and mailserverversion is the name of the software that the remote mail server is running. Depending on the software the target server is using, you may not see a prompt that waits for you to enter a command. You’ll only see a flashing cursor.

The first thing you’ll do is send the helo domain command to the mail server to begin communication with the mail server where domain is the name of the domain you’re coming from. Don’t panic if the mail server doesn’t echo the commands you type. Just type the commands slowly and press [Enter] when you’re done. If you enter the command correctly, the server will respond 250  OK.

Next, you’ll simulate an e-mail message by typing MAIL FROM:<emailaddr> and pressing [Enter] where emailaddr is the e-mail address you want to send a test message from. Make sure you include the : and the <>’s. If you enter the address wrong, the server will respond 553 Malformed Address. If everything’s ok, the server will respond 250 OK - MAIL FROM <emailaddr>.

After that, you must enter the e-mail address the test message is going to. You’ll do that by typing RCPT TO:<emailaddr2> where emailaddr2 is the e-mail address of the person you’re sending the message to. If everything works properly, the server will respond 250 OK - Recipient <emailaddr2>.

Next, enter in your test message. Start by telling the mail server that there’s a message coming by entering the data command and pressing [Enter]. Enter your message and then conclude it by pressing [Enter].  The mail server will then either send the message to the user’s mailbox or respond with an error message. The actual error message will vary depending on what the problem is. To close the Telnet connection, type quit and press [Enter].

As you’re using Telnet, the responses you receive will help you identify why mail from your server is unable to reach the target server. The target mail server may respond with several errors, including:
  • Mailbox has exceeded an allowed size
  • Relaying prohibited
  • User doesn’t exist

Based on the responses you receive, you can then form a strategy to resolve the problem.

Help! Nothing works
If you get no responses from the target mail server, there are several things to consider: First, and most simply, the target mail server may be down. Second, the company you’re sending e-mail to may be transitioning from one ISP to another, causing mail to be lost in the process. Third, the company could be transitioning from one mail server to another one that is using a different TCP/IP address from the one listed in DNS. If this is the case, the mail you’re trying to send will never find the appropriate server until the DNS server is corrected.

One size fits most
You now have the tools to identify where the problem is when you can’t send e-mail from your company to another. The best suggestion I can give you is to be consistent in your approach to this type of problem. Only in a few cases has using Nslookup and Telnet together not given me all the information I needed to identify the problem that prevents mail from reaching its destination. When those cases arose, I used a protocol analyzer to look at the conversation between the two mail servers and was able to find the problem.

Editor's Picks