Web Development

DNS: Painful reminders of how important it is

Having a healthy and accurate DNS architecture is critical to the operation of any network, especially the Internet. Michael Kassner shares some thoughts on the crucial, yet often over-looked protocol -- especially one hard-learned lesson garnered after troubleshooting a client's DNS problem last week.

Michael Kassner The Domain Name System (DNS) protocol isn't essential in networking, but it sure makes everyone's life a great deal easier. Without it, we'd have to use cumbersome numeric IP addresses (XX.XX.XX.XX) instead of Fully Qualified Domain Names (FQDN) like www.example.com. I don't know about you, but that would be frustrating for me, as I tend to transpose numbers for some unknown reason. The relationship between an IP address and FQDN is also important to network engineers. They use it to see if DNS is working properly. If a FQDN doesn't resolve, but the FQDN's associated IP address works, it's a pretty safe bet there's a DNS-related problem. Sounds simple, but the relationship is easily forgotten when an entire facility is down and the CEO is upset that e-mail isn't working.

I know I said that DNS isn't essential in modern networks, but that's only on a very fundamental level. Everything is set up to use DNS, so any kind of disruption to DNS will immediately create problems, which gets the attention of network engineers fast; as everything is down, nothing works. Or so I thought.

DNS problems can be sneaky

A client of mine called, and I could tell right away it wasn't a social call. In a panic, he mentioned that no one was receiving any external e-mail (PoP3 or IMAP). Internal e-mail worked and so did outgoing e-mail (SMTP). Web browsing worked and so did VPN-accessed applications. I started to get a sinking feeling about the Exchange server. So I immediately set up a VPN tunnel into the facility. After running a few tests and checking the error logs, I was a bit perplexed as to why internal e-mail would be working as well as outgoing e-mail, but not incoming e-mail. Spam filters? Hmm, I disabled the spam filters and no change.

After running several more tests, I started to understand what was happening. The internal DNS servers were working properly, allowing the outgoing e-mail and Web browsing to resolve FQDNs. What about external incoming...? Before I could finish that sentence, I had an ah-ha! moment. I went to my normal Internet utilities Web site, quickly asking for the WHOIS and DNS records for the client's domain. There it was, the canonical name wasn't resolving into an IP address. The name server and MX records were missing as well. Okay, why was that? Thankfully another ah-ha! moment. The client recently instructed the ISP to discontinue hosting the unused ".net" domain name. A quick call to the ISP confirmed my suspicions. The ISP thought my client wanted all hosting to stop and that's why the records for the ".com" domain were missing as well. After significant begging, the ISP kindly rushed a work order through. Within the hour, external e-mail started arriving again.

DNS is important

I hope my example points out how utterly reliant we are on having DNS work properly. Most network engineers I know constantly fret about the health (up-to-date software) and accuracy (DNS records) of their DNS servers. Sure DNS server health makes sense, but why is accuracy even a consideration? Well, funny you should ask.

There's a well-known flaw in the DNS protocol that allows attackers to replace valid DNS content with entries of the attacker's own choosing by using a "cache poisoning attack." The poisoned DNS server will then offer incorrect content to DNS queries. This technique, for example, could redirect a Web browser to a malicious Web site that mimics the official Web site. After the redirection, any number of attacks could be made on the unsuspecting device that made the original query.

DNS cache poisoning just got easier

Most network professionals weren't that concerned about cache poisoning as it's non-trivial to implement. Last week -- well, at least to the public (I'll explain later) -- that all changed. New research has led to exploits that leverage the same old vulnerability, but in a unique way -- by improving the effectiveness of cache-poisoning attacks and simplifying the attack methodology.

Dan Kaminsky (director of pen testing at IOActive) reported that he accidentally uncovered a method that would allow an attacker easily to disrupt the Internet by attacking DNS servers. For a concise explanation about the vulnerability, listen to the NetworkWorld podcast "DNS Flaw-Fix Hype Addressed" where senior editor Denise Dube interviews Kaminsky about his findings. In addition, Dube and Kaminsky talk about the events that took place after finding the flaw.

DNS fix elicits controversy

The solution developed to rectify the vulnerability was rather unprecedented and to say the least interesting. It seems that the major parties concerned with DNS (a total of 16 entities, including ISC, CERT, Cisco, Sun, Microsoft, and major ISPs) acted in unison and under a cloak of secrecy to create the fix over the past several months. Hence my earlier comment about the vulnerability only being made public last week. For more about the controversy, please check out the TechWorld article "Hackers Gang Up on Kaminsky over DNS Flaw." David Dragon, a DNS researcher from Georgia Tech, adds validity with the following comment in the referenced TechWorld article:

"The issue is urgent and should be patched immediately. With sparse details, a few have questioned whether Dan Kaminsky had repackaged older work in DNS attacks. It is not feasible to think that the world's DNS vendors would have patched and announced in unison for no reason."

The controversy deepens, because Kaminsky will not publish any details of the DNS flaw until his Black Hat presentation next month. Some say waiting until Black Hat is grandstanding, but Kaminsky makes a valid argument that waiting until then will allow engineers almost a month to get vulnerable DNS servers patched. In the same vein, security researchers are concerned that Kaminsky didn't ask for any peer review before announcing the vulnerability. In his defense, Kaminsky has briefed a few well-known security researchers (Thomas Ptacek of Matasano Research and Paul Vixie of Internet systems Consortium), and they readily admit that his findings are correct.

Final thoughts

I personally admire Kaminsky. He could have very well taken advantage of his finding and sold that information to the highest bidder. As noted in the following quote from the NetworkWorld article "Major DNS Flaw Could Disrupt the Internet":

"Jeff Moss, founder of the Black Hat conference, applauded Kaminsky for treating the DNS discovery he made with a sense of responsible disclosure, rather than selling the information to the highest bidder, a practice growing increasingly common. If he had decided to sell it, he would have made hundreds of thousands of dollars."

My ultimate goal with this article is to raise awarenes of the critical nature of DNS. I sincerely hope that everything possible to harden DNS is being done. Without it, our Internet life as we know it would be in dire straits.

-------------------------------------------------------------------------------------------------------------------

Michael Kassner has been involved with wireless communications for 40 plus years, starting with amateur radio (K0PBX) and now as a network field engineer and independent wireless consultant. Current certifications include Cisco ESTQ Field Engineer, CWNA, and CWSP.

About

Information is my field...Writing is my passion...Coupling the two is my mission.

104 comments
Michael Kassner
Michael Kassner

I thought I'd have time to talk about the DNS checker in my next article about the DNS vulnerablity. It appears that isn't the case though. So I then thought the members may want to know that Kaminsky has an application on his website that can be used to see if your DNS servers are vulnerable. http://www.doxpara.com/ If so, the OpenDNS servers are not vulnerable to this attack and maybe a good solution. http://www.opendns.com/

Michael Kassner
Michael Kassner

I guess the meeting was a big deal. I didn't even know about it until it was almost over. I tried to see if I could go and listen, but it was only for invited guests. I suspect it's still early, but I haven't heard any outcomes yet either.

seanferd
seanferd

I have found your links and will follow them.

seanferd
seanferd

I'll be right over there. Thanks, Michael.

seanferd
seanferd

I'll be heading right over there.

seanferd
seanferd

I kept checking, and although AT&T has been mentioned as an ISP that was "on board", they still hadn't patched by the time the exploit code was out. So I went the OpenDNS route (very easy). I had used it in the past, but not since I went DSL. Folk, this is also the time to make sure you have a secure password on your gateway/modem/router. Two birds with one stone.

seanferd
seanferd

There is always the key-signing key issue between ICANN and Verisign, although Verisign has oh so graciously come to suggest sharing key signing among all root authorities.

Michael Kassner
Michael Kassner

Cookies and redirection at work once again. Thanks Sean, great article. I thinks it's important enough to write about.

seanferd
seanferd

First, I added that comment after already reading the second article. It had to do with the lateness of the email alert I had received, but the sentiment still stands. Second, thumbs to you both! Unfortunately, the only way to bestow a thumb is when if you've received a reply to a Question thread that you started. Otherwise, there is just no way to do it. This is why I sometimes add the "conceptual thumb" to my comments.

Michael Kassner
Michael Kassner

I consider both of yours as well as the other member's opinions more important. I'm pretty sure the thumbs up is for the member that provides the best comments, so both of you should be getting them then.

JCitizen
JCitizen

but I've never got around to figuring out how to bestow them. The help files here at TR don't help.

JCitizen
JCitizen

I have my router shutdown from remote administration, but I know what you mean; the crack can come from within the LAN also with these kind of exploits going around. I listened to the MP3 broadcast on the issue and was enlightened - thanks for the links! I can't help thinking that my ISP isn't answering the phone right now because of this. I've been getting a lot of legitimate pages with certificate errors the past two days. I just thought it was a fluke because the pages I was switching to were within an SSL session and my personal information was already on them, and correct. VERY scary!

Michael Kassner
Michael Kassner

Good thought, Sean. OpenDNS is a premier site that has a great history and rep.

JCitizen
JCitizen

go deeper into network traffic analysis. I need to get in some reading so I can quit wasting people's time picking their brains! I sure do appreciate Michael's and your efforts. Both of your brains probably feel like pin cushions some times. I know TR sells things like this but they have so many titles I never know which one to plunk my money down for.

JCitizen
JCitizen

problem. XHTML Looks like it would be a good candidate for web standards if the requirements are tight; could force all developers to fall between the lines?

JCitizen
JCitizen

what I'm used to but a piece of cake to operate of course. This will make my Army buddies in Iraq happy as they don't like doing anything outside of https:\ I think I'm already liking it! =)

seanferd
seanferd

You can pipe gmail through your favorite email client, if you like. They don't make it obvious, or anything, but you do have that option. I did it when I decided I needed a separate account for work related communication.

JCitizen
JCitizen

I wonder sometimes if your exchange server is dumping some of your mail? I would be glad to resend it, if you like.

Neon Samurai
Neon Samurai

I test the same way. My personal network's mandate is to not see C&A pop any uname/passwd off my network. It's a quick, easy test for your local Windows machine or you can "place" it between your two *nix machines. That's my quick test. For the more comprehensive tests, grab your tcpdump, wireshark or prefered industrial strength tool.

Michael Kassner
Michael Kassner

To be safe, I'd sniff the traffic upstream of your computer to see if the entire data steam is encrypted. Also JCitizen did you get my email?

JCitizen
JCitizen

It will take me a while to get to using it but I like the extra security. Thanks yet again Neon!

JCitizen
JCitizen

my buddy in Iraq has been trying to get me to switch; guess I'd be an idiot not to!

Neon Samurai
Neon Samurai

Gmail is defaulting to https so that should encapsolate the entire session. No more sniffing vuln, just MITM if someone can catch your session initiation.

JCitizen
JCitizen

at least that would make me feel better. This surf jack exploit; does using SSH vs SSL make any difference? Also I noticed my hotmail went totally https:\\for logging in some time back even though I used to have to manually select it everytime. Of course the entire email session is secure with gmail now? I thought I remembered reading that.

-Q-240248
-Q-240248

No one has hijacked any domain by dns cache poisoning, and if they do, you can pretty much be sure it wouldn't be there long...

JCitizen
JCitizen

Hope you can open the WordPad attachment..The visual route gives enough information that one probably doesn't need WHOIS, but I looked them up anyway just to confirm.

Michael Kassner
Michael Kassner

It's an interesting piece of research. I PM'd you. I also would like you to do a Whois to find out what the exact names of the DNS servers your ISP is using as it will state that on the report.

JCitizen
JCitizen

on the map but not a land address; we are talking about the Visual Trace of course, but although the city is obvious where the "unknown" hop was, there is no IP on that tool, I'll try the cmd tracert and see what happens but it won't use the same source as you already know. I'm purposely leaving out the details for two reasons; 1. I don't want to give away the location of the servers for their security. 2. I don't want to give away any information that would in any way give someone enough information to triangulate my location. I have to keep security on that for various reasons; but I was hoping to shed light on the fact that perhaps one's own ISP doesn't rely on localized DNS during this patch/transition phase. They may be using service from hundreds of miles away, completely out of their service area. These tools might not reflect the closest DNS server that could affect the tester(IT person using the tool). If you would like exact information Michael; I will email you, because I trust you, but I don't like bothering any other member unless they email me first. I have a hang up about interrupting other professional's busy schedules.

Michael Kassner
Michael Kassner

JCitizen, you still didn't confirm whether you have determined the physical location of the name servers from other than a trace. That's where I'm getting a bit confused. If you know that then you also know if you are having an issue or not with the unknown name server. If you have told me, I'm sorry for missing it.

JCitizen
JCitizen

That is the best trace tool I've EVER used. Sheeze! Thanks seanferd!!!! I didn't even need to look up my ISP IP with that one. I just watched each hop right in front of my eyes! Symantec used to have a similar one with NIS; but it wasn't this good! From the map in this tool I discovered that DoxPara gives me results from the [b]27th[/b] hop! Which goes clear to California and almost all the way back. The one you gave before put it at the forth hop which was still far away but actually took a shorter journey. Sorry for the stilted response as I don't want to give my ISP away on this discussion, or my location. I would be glad to finish this with either of you if your just curious, by direct email communication. Is it any concern that the fifth hop was "unknown"; this gets scarier to me, but I'm just naturally paranoid about IT security.

Michael Kassner
Michael Kassner

Thanks for sharing Sean, I definitely bookmarked it. I like how they pull of a web-based trace.

Michael Kassner
Michael Kassner

I like that website. I was wondering how they were going to run a trace. It's the first one I've seen that runs the trace through their own server. Kind of neat. I definitely have it bookmarked. Thanks Sean.

Michael Kassner
Michael Kassner

I like that website. I was wondering how they were going to run a trace. It's the first one I've seen that runs the trace through their own server. Kind of neat. I definitely have it bookmarked. Thanks Sean.

seanferd
seanferd

http://www.yougetsignal.com/tools/visual-tracert/ This is what I originally thought you were doing. I don't know if I've used this particular site before, but I've used others just like it (usually for locating the origins of spam or forum spam, along with other tools).

Michael Kassner
Michael Kassner

JCitizen, I hope hopping is a good thing and not that you're under pressure. I was a bit confused about your Whois answer. Where you able to resolve the IP into a location?

JCitizen
JCitizen

I never got around to finding my local DNS IP so I haven't tried it yet. Thing are really hopping around here though; I've been greatly distracted. Sorry for the slip! I've been seeing the IP and looking it up through whois.

seanferd
seanferd

Is that farther away by number of hops then? I had assumed earlier that you'd done some visual traceroute to see the actual geographic location of the server. Is that me sowing confusion again? :)

JCitizen
JCitizen

But it was better at the randomness. I just hope that is the one my ISP actually uses. Hey, if they make me mad about my HDTV reception, I'll still have to "go down there" HA! ;)

seanferd
seanferd

I like it, and it's a bit more visual and emotional, which is an aid in getting some people to grok the importance of using a patched (or already secure) DNS server. I'd read some folk's were getting a differential in their results between various DNS tests. I hadn't checked against the link for another test I'd seen, somewhere...

seanferd
seanferd

I'd hate to be telling anyone stuff that is entirely wrong. :) The good news is that AT&T finally patched the servers in use for my area, and another popular cable provider has as well. That made it easier for me, as I could slack off on my continuous evangelizing of "Check my DNS". ;)

JCitizen
JCitizen

It all seems like a CCNA review to me! Hopefully this vulnerability doesn't extend to the ISP local. They probably won't discuss the issue either,and I can't say as I blame them!

Michael Kassner
Michael Kassner

Sean, I believe your interpretation is correct. In either case the process is as you suggested. I think the main purpose of using automatic discovery is to have that be one less thing for people to be concerned about. Also, in my experience ISP's change their DNS server IP addrs more often than thought. When that happens the ISPs would get a rash of phone calls if everyone had static IP addrs configured. JCitizen, It's my understanding that good ISP practice is to: 1. Have the name servers be on different subnets. 2. Have a secondary name server that is not in the same physical region. Both are important to maintain a physical presence in all possible situations.

JCitizen
JCitizen

my ISP has two servers for redundancy, but I just set it on the same settings I would have used for a DSL modem or any other client in this area. Inside my LAN the units are set to automatic, of course. Perhaps the tool only checks the closest high level DNS server to the geographic location of the client? Now you got me going; I'll have to do a ping/tracert to the IP addresses there and see what happens now. I've been stretched for time - going nuts learning how to setup HDTV in my home, and looking for a good PC media center hardware configuration.

seanferd
seanferd

For my home router, I can "Obtain DNS information automatically", or "Manually configure DNS information". My manually configured primary and secondary DNS server IP addresses would definitely be static, but I assume, perhaps incorrectly, that when automatically obtaining the ISPs DNS IP, the ISP could change that address, providing they have another server to which to switch. Have I conflated manual-automatic with static-dynamic here? These phrases seem to be used in the same manner when referring to obtaining or setting one's own IP address, but perhaps I've stretched the similarity too far. edit: to for too (egad!)

JCitizen
JCitizen

especially if they are doing it utill they patch the local one. I think that is a level 3 DNS server; my foggy brain has slipped on the structure of the cloud, so I'll refrain from making comments on that part of it.

Michael Kassner
Michael Kassner

Sorry Sean, you can't be that cryptic. You will have to explain yourself for us slow types. I'm used to networks having static DNS servers.

seanferd
seanferd

That's an interesting idea. Just a thought: Perhaps your ISP has swapped you to a patched server while one closer to you is being updated. That is, assuming you haven't been using the same server all along. I don't think mine had ever changed, and the config was set to dynamic.

JCitizen
JCitizen

That is if you trust "Shields UP" at GRC. I know some routers will flash port 113 as closed for some ISPs that use it for management purposes; but I gave away the last router I had because it was doing that. My argument was that if GRC could do it, so could the cracker! Netgear finally came out with a new firmware update for that machine; but I haven't met with the guy that has the router yet to see if the flash file stealthed the port. This took Netgear almost 5 years to come up with an update! Ridiculous!

JCitizen
JCitizen

Your name server, at XXX.244.4.227, appears to be safe, but make sure the ports listed below aren't following an obvious pattern (:1001, :1002, :1003, or :30000, :30020, :30100...). -------------------------------------------------------------------------------- Requests seen for 3xxx8a7c78c2.doxdns5.com: XXX.244.4.227:43635 TXID=2530 XXX.244.4.227:50514 TXID=64659 XXX.244.4.227:40217 TXID=56274 XXX.244.4.227:42639 TXID=48946 XXX.244.4.227:45866 TXID=43763 I X'ed out the IP address and "as seen" for security reasons. But this server is a long way from where I live. I always thought your ISP used DNS in the first hop? This all may be beyond my level of profession! :(

JCitizen
JCitizen

and I say that because I logged on and my personal information was already there and correct. As I was navigating the site to go to another account information page IE 7 popped a certificate error on me. I could see the URL was correct, and hoping this was a fluke I blew through the warning and there was the page I wanted with the correct information. Listening to Kaminsky, made me think of this seemingly innocuous error, and I decided to contact my ISP site. The first time I called the site administrator was out for the day; and now they won't answer my calls or messages. I haven't emailed them yet, because I'm kinda busy this week. It is worrisome though!

Michael Kassner
Michael Kassner

I listened to Steve Gibson's Security Now podcast weekly and he brought up an interesting oddity about perimeter routers. You mentioning about router configurations triggered my recalling it. A good thing to check is if port 80 is open on external interface of the router, I guess several brands of routers still advertise WAN side remote access even though that option was disabled in the configuration of the router. I don't have any brands or names though. Just a thought.

seanferd
seanferd

Like Kaminsky's at doxpara.net? I kept checking, as AT&T was supposed to be "on-board" with this, but was still showing as unpatched. Routers, etc.: Yep, I disable everything not in use, I also disable services and devices for remote help/ admin., Term. Serv. Dev. redirector, etc. My thinking was, if yer gonna go into your router to switch DNS servers, you may as well check other things to make sure they're locked down. I disable wireless, for example, as I don't use it at all. (It was a bit of a fight, that. Couldn't connect over Ethernet cable for some reason, and a nearby cordless phone may have been a part of the problem as well. Still, you need to set up wireless while cabled, so I can't really grok what the issue was.)

Michael Kassner
Michael Kassner

Did you get sent to a bad site because of DNS errors? Would it be possible to go into more details? If not I understand for real.

Editor's Picks