Intrusion detection systems (IDSs) are becoming increasingly important to
every company’s defense in depth strategy. An IDS acts to alert you that a
possible attack is in progress and informs you about characteristics of the
attack. In addition to providing information about attacks in progress, an IDS
can provide you information about reconnaissance efforts make by intruders in
preparation for an attack. A more advanced form of IDS, called IPS (Intrusion
Prevention Device) will initiate countermeasures to stop the attack from

As useful as an IDS/IPS can be in
proactively and reactively protecting your network, the IDS will only be as
useful and effective as your implementation allows it to be. There is a
tremendous amount of confusion in the IDS space today regarding what an IDS
should detect, how the IDS should detect, and what an IDS actually is.

There are several common blunders, or implementation errors, that
administrators make when setting up their IDS/IPS. These can prevent you from
getting the level of protection that you need and expect from your IDS software
or device.

Distinguishing between IDS types

In this article, we’ll focus on blunders that
apply to two different types of IDS approaches:

  • Network
    IDS (NIDS)
  • Host
    IDS (HIDS)

A NIDS is a device (often referred to as a “sensor”) placed
somewhere inside or in front of the corporate network that listens to all
traffic moving by the device. A NIDS can be a dedicated hardware device, or it
can be implemented as a software add-on to a general purpose network operating
system. A pure HIDS doesn’t look at the network traffic; instead, the HIDS
analyzes log files from a number of system services and network applications
running on the host. Some IDS software incorporates functions of both the NIDS
and the HIDS.

Both HIDS and NIDS can use one of two primary approaches to intrusion
detection: anomaly detection and signature detection.

  • Anomaly detection attempts to profile normal
    behavior for the environment in which the IDS is placed, and then calls out
    activity that falls outside this norm. Anomaly detection is akin to what you
    see with Bayesian anti-spam applications.
  • Signature-based IDS systems include a database
    of patterns that the IDS matches against the environment on which it’s located.
    Signature-based IDS is similar to what anti-virus applications use to detect
    viruses on host systems or over the network.

While there are a multiplicity of implementation errors that can be made
when deploying an IDS, there are five that we see most commonly. These are:

  • Ignoring frequent false positives
  • Avoiding IPSec to support NIDS
  • Monitoring only inbound connections
  • Using shared network resources to gather NIDS
  • Trusting IDS analysis to non-expert analysts

In the following sections, we’ll take a look at each of these, their
ramifications, and how you can be sure to avoid them when you set up your IDS.

Ignoring frequent false positives

Usually, when organizations adopt a new IDS they will turn the IDS system on
to detect any and all possible exploits. In other words, they turn the IDS on
to the level of its highest sensitivity. This seems like the logical thing to do; after all, you want to catch
every intrusion, don’t you? Unfortunately, while this configuration enables the
IDS to detect a greater number of possible attacks, it also leaves the system
open to a greater number of false

False positives are the bane of the IDS world. There are two main errors you’ll
encounter when working with IDSs: false positives and false negatives. A false
positive occurs when the IDS reports a potential attack but there is actually
no attack in progress. A false negative is when the IDS fails to report an
attack when an attack is in progress. False negatives indicate a failure in the
IDS itself, while the false positive shows the IDS is working exactly as it is
configured to work.

The problem with false positives is that IDS admins begin receiving hundreds
or thousands of alerts per day, overwhelming them and leading them to ignore it
when the IDS reports a real attack because it has “cried wolf” so
many times in the past. A good example of how this happens is when you’ve
configured your IDS to liberally interpret port scans as attacks. Ports scans are going on all the time, and
most of them don’t result in actual intrusions. If you configure your IDS to
report every port scan, you’ll probably soon find that you’re ignoring not only
the port scan alerts, but any other alert that the IDS sends you. That’s human
nature–but it can have disastrous results.

You should not ignore false positives or become complacent about them. The
IDS is doing its job when it reports an event. You should investigate the
events reported to you by the IDS by analyzing the appropriate packet traces or
host logs, and also correlate NIDS and HIDS data. If your analysis determines
that the IDS is indeed showing too many false positives, then you should
dial-down the sensitivity for that alert, or turn it off completely.

Avoiding IPSec to support NIDS

We tend to think of encryption as the ultimate security measure. For
example, many network administrators consider VPN connections to be secure
connections because the data inside the tunnel is encrypted by MPPE or IPSec.
The problem is that many people confuse tunnel encryption with access control.
A VPN connection makes the communications between the two VPN endpoints
private, but once the data moves outside the VPN endpoints, it is no longer
private and thus not secure.

Another common problem with encryption over the wire is that network
security devices cannot perform stateful application layer inspection on the
contents of the encrypted data stream. This is a major problem in the firewall
world, where modern firewalls, such as the ISA Server 2004 firewall, can
perform both stateful packet and
application layer inspection. The problem is that advanced firewalls can only
perform stateful packet inspection (not application layer inspection) on
encrypted communications because the firewall does not have access to the
encrypted application layer contents; thus, attackers can easily hide their
exploits from the firewall’s application layer inspection mechanisms by using
encrypted tunnels.

As it is with firewalls, so it is with NIDS. The NIDS needs to be able to
listen to all traffic moving past its interface and compare that traffic with
its rules for legitimate and illegitimate communications. When the NIDS
encounters encrypted traffic, the only analysis it can perform is packet level
analysis, since the application layer contents are inaccessible. Given that
exploits against today’s networks are primarily targeted against network
services (application layer entities), packet level analysis ends up doing very
little to protect our core business assets.

This issue comes into play in environments that have an established NIDS
infrastructure and are considering using IPSec for domain isolation.
IPSec-based network access controls provide a powerful mechanism to control
which hosts are allowed to communication with which hosts on the corporate
network. However, one of the key features of both encrypted and non-encrypted
IPSec communications is that only the source and destination hosts are able to
access the content of the communications.

This creates angst for the “network guys” who run the NIDS on the
network, because they believe that IPSec reduces their level of network
security since their NIDS devices can’t assess the full nature of the IPSec
protected communications. While they are correct about hiding the nature of the
communications from the NIDS, they are unequivocally incorrect about this
reducing the network’s overall security posture when you use IPSec.
Implementing IPSec for domain isolation can only improve a company’s overall
security posture.

The solution is to use HIDS in conjunction with NIDS. The HIDS sensors are
placed on the IPSec endpoints and can detect potential attacks. The risk of
attacks is lower when using IPSec access controls, so the end result is that
HIDS data will be easier to interpret because of the lower volume of data. In
addition, there will be many fewer false positives.

Monitoring only inbound connections

There are two general approaches to firewalling and access control: control
all traffic moving inbound and outbound through the firewall, or control only
the traffic moving inbound and allow all outbound traffic to have a “free
pass”. Unfortunately, even in today’s security-conscious world, many
organizations don’t subscribe to the principle of least privilege and don’t
enforce outbound access control that enable users and applications Internet
access only for those resources they require to accomplish their duties.

The same issues are extant for IDS systems. Some IDS admins are very
perspicacious about monitoring external connections, such as DMZs in front of
the corporate firewalls, or at the firewall interfaces themselves, but pay no
attention to the traffic emanating from corporate network hosts, either
outbound to the Internet or to other segments on the corporate network.

Make sure that you place NIDS sensors throughout the organization. The ideal
situation is that a NIDS is connected to a spanning port on all network
switches. However, if that is cost prohibitive, you should at least place NIDS
at the chokepoints on the corporate network. This enables you to monitor
outbound and inter-host communications on your network. This is an absolute
requirement on today’s networks because of the prevalence of network worms and
other automated attacks.

Using Shared Network Resources to gather NIDS data

Often NIDS administrators will deploy NIDS sensors on single NIC or
multihomed devices on the network that have one or more connections to
production network segments. The implication of this configuration is that the
NIDS sensor will send data captures over the same interface on which the sensor
is sensing.

This is a less than optimal security configuration because the same
interface is used to collect data and report it to centralized reporting
databases. An attacker can leverage this network configuration to do one of two

  • Disable the IDS and prevent it from sending an
    alert, or
  • Intercept the data being transferred to the
    reporting database before it reaches the database and change the nature of the
    data, which could potentially be done with a man in the middle attack.

You can solve this problem by using multihomed NIDS devices and connecting
the non-listening interface to a dedicated monitoring network. The dedicated
monitoring network connects only the NIDS devices, the reporting database servers,
and the monitoring workstations. In the event that the NIDS devices are
attacked, an alert can arrive from the monitoring network interface so that you
can respond in a timely fashion.

Trusting IDS analysis to non-expert analysts

This one might seem like just a matter of common sense, but it’s a situation
we see all too frequently. In this scenario, a company has already spent quite
a bit of money for NIDS, HIDS or both, and they expect the current staff to be
able to implement, manage, and interpret the IDS alerts and comes up with the
appropriate responses to these alerts. Often, this belief is based on what the
IDS vendor told them in pushing the “turn key” nature of his product.

While the IDS device can do the “heavy lifting” in terms of
listening for network traffic, analyzing host log files, and matching the
findings against rules configured on the IDS device, an experienced intrusion
detection specialist is required to give meaning to many of the events reported
by the IDS.

That’s because once the potential threat is detected, an intelligent
operator must be able to review the alerts for their validity. Many times
legitimate communications are interpreted by the IDS as attacks, and if the IDS
operators blindly accept these as actual attacks, they could end up denying
service to innocent hosts on both internal and external networks.

The IDS analyst must be expert in TCP/IP networking, analysis of networking
logs and packet traces, and also extremely well informed about the network
services and applications running on the HIDS enabled devices. This level of
knowledge is required to correctly interpret the IDS alerts to confirm their
validity and then execute the correct intrusion response.