Tech Tip: Prevent service masquerading on your network

Depending on your network security configuration, and specifically the Internet perimeter, your organization could have glaring holes in its security, and you may not even know it.

By Jonathan Yarden

I recently read an article on CNET Networks' that discussed an Internet security risk that few organizations consider. In fact, the security hole is so large—and the concept so obvious—that I'm slightly embarrassed I didn't consider it myself.

The article discussed a security researcher's presentation at the recent DEFCON 12, a hacking conference held in Las Vegas a few weeks ago. Dan Kaminsky, a security researcher for telecommunications firm Avaya, discussed a security hack that uses data transferred by DNS servers to conceal information in network communications.

Hackers can use this vulnerability to bypass organizations' security measures, including firewalls and intrusion detection systems (IDSs), which typically disregard DNS data because they assume it's safe.

Before we delve into the details of this vulnerability, let's briefly review typical firewalling concepts. A firewall on the perimeter between the Internet and internal networks generally focuses more on filtering and blocking data from the Internet than from the internal network.

While some companies go a step further to restrict outbound traffic, the security focus is generally on stopping unwanted IP traffic from coming in—not going out. And even though firewalls that inspect data in both directions exist, they don't inspect all possible services, networks, and hosts.

That means it's entirely possible for a hostile program on an internal host to masquerade its activity by using a well-known TCP or UDP service. Depending on how a company restricts traffic and monitors its internal network, its firewall and IDS system could easily fail to notice traffic from a masquerading service.

An even more disturbing tactic would be to subvert legitimate TCP or UDP services into "bouncing" data on top of seemingly legitimate requests. Only careful inspection of the data packets would reveal the masquerade.

While the idea of covertly passing data isn't new, Kaminsky demonstrated software that manipulated DNS to allow internal firewalled hosts to send encrypted data that's indistinguishable from legitimate DNS traffic outside of an otherwise secure network.

While defending a corporate network from this type of activity isn't currently possible, some methods exist to prevent more general types of service masquerading. To begin, configure firewalls with rules that restrict specific UDP and TCP services originating from internal networks. This is already a common practice with transparent HTTP content filtering proxies, which intercept and funnel HTTP requests to make sure employees keep their eyes on legitimate work issues.

Most enterprise firewalls have features to block outbound requests or force well-known UDP and TCP services to specific internal hosts and deny other traffic. This limits the ability of internal hosts to use arbitrary UDP and TCP service ports to access the Internet. It also prevents Trojans on internal hosts from masquerading as well-known UDP and TCP services and bypassing your perimeter security devices.

For example, don't allow SMTP, HTTP, or FTP traffic to travel from the Internet to a DNS server. If a hacker compromises the DNS server, he or she can install and then use these services.

Likewise, don't allow the DNS server to pass traffic to or from anything other than UDP or TCP port 53. And where possible, use a combination of perimeter firewalling and host-based firewalling to protect internal networks from other internal hosts in the event of a problem.

Windows XP and Windows Server 2003 both sport easy-to-use host-based firewalling, and Windows 2000 can use Sygate Personal Firewall in a similar manner. All of the free UNIX variants have excellent firewalling capabilities as well.

Depending on your network security configuration, and specifically the Internet perimeter, your organization could have glaring holes in its security, and you may not even know it. But by locking down hosts and networks in this manner, service masquerading can cease to be a risk to your enterprise.

Jonathan Yarden is the senior UNIX system administrator, network security manager, and senior software architect for a regional ISP.

Editor's Picks

Free Newsletters, In your Inbox