Security

Use security log management to monitor network activity


Most of us have the recommended layers of protection in place.  Firewalls, email filtering, IDS, and IDP systems protect the perimeter and critical network segments.  Hardened servers, anti-malware and carefully managed access controls protect individual devices as deperimeterization increases.  But how effective are these controls?  Is there activity, successful or unsuccessful, that indicates that one or more controls might have failed?  This kind of visibility is provided in large part by security log management.

What is Security Log Management?
Logs containing information relevant to security management are generated by many sources, including:

  • Firewalls
  • Intrusion detection and prevention systems
  • Anti-malware systems, especially centrally managed solutions with aggregated reporting
  • Operating systems
  • Switches
  • Routers
  • Workstations
  • Applications

Many organizations ignore the logs until there’s a security incident.  However, preventive security measures require a daily review of information from business critical sources.  This type of review can help identify (Karen Kent and Murugiah Souppaya, NIST SP 800-92, September 2006):

  • Security incidents
    o Password hacking
    o Large numbers of login failures
    o Malware attacks
    o Port scans
    o Denial of service attacks
    o Excessive errors on network devices
  • Policy violations
  • Fraudulent activities
  • Operational problems
  • Regulatory compliance issues

The challenges to log review can be overwhelming to many businesses.  Logs are continuously growing, are located in many silos, and the staffing and skills necessary to make sense from all the information collected is unavailable.  Security Log Management helps with the process of aggregating, correlating, and reacting to information captured in logs across an enterprise.

Creating a security log management process
The first step in creating a log management process is the creation of a policy.  The policy should define the objectives the organization wants to meet by managing log information.  Supporting standards and guidelines are necessary to ensure policy compliance.  According to Kent and Souppaya, the following issues should be addressed:

  • Log generation
  • Log information transmission
  • Log storage
  • Log analysis
  • Log disposal

When deciding how and when to generate logs, security managers should carefully select the information required.  Data contained in the logs should match those needed to hit management objectives as defined in the policy.  Building a table to list and define log data elements, as in Table 1, can be helpful (Amrit T. Williams, “Define Application Security Log Output Standards”, Gartner ID G00139205, 4 May 2006).

 Table 1

Williams defines additional tables in his paper.

However, all logs are not created equal.  Daily reviews of logs related to an organization’s most critical systems are required.  However, Kent and Souppaya write that less than critical system logs can be on a less stringent schedule.

Finally, a tool should be selected that will pull all the log information into a central place, compare entries from various sources to identify relationships, and provide a portal for viewing the results.  The volume of log information in most organizations makes manual log analysis impractical.  Implementing the right log management solution, whether in-house or from a managed security services provider, is the best way to ensure log analysis provides the best picture of network activity.

Regardless of the solution selected, there are some general best practices that can help ensure success (John Howie, “Security Log Collection”, Windows IT Security, 16 Oct 2006), including:

  1. Secure your log repository.  Log data integrity is important during litigation discovery or when attempting to reconstruct suspected security incidents.
  2. Ensure the clocks on every system from which logs are collected are synchronized.  This is critical for effective event correlation activities.
  3. Frequently revisit the log data elements collected.  As security policies, standards, guidelines and threats evolve, so should your log management process.
  4. Calculate how much storage you’ll need to store log information.  This is closely related to your log disposal policy.
  5. Check the log collection system frequently to ensure it’s working correctly. 

The final word
Log management is an essential part of any security program.  Without the visibility it provides, a security manager lacks the ability to proactively address potential weaknesses in security controls—while reacting blindly to security incidents.

About

Tom is a security researcher for the InfoSec Institute and an IT professional with over 30 years of experience. He has written three books, Just Enough Security, Microsoft Virtualization, and Enterprise Security: A Practitioner's Guide (to be publish...

16 comments
lbert33
lbert33

All agreed that log review is critical and there are numerous tools to assist. question - without any tools, what are the items to look out for firewall logs? keywords? strings?

ctp
ctp

Its great to monitor log files and its great to agregate them, but what about security of the files themselves? Its real easy to edit a log file especially when you put them all in one place.

Tom Olzak
Tom Olzak

How do you manage your security logs?

ctp
ctp

Its not possible to select some "generic" set of terms/items. Each firewall vender does it differently. Any tool that you use should support different "patterns" for different devices and they should provide you with a place to start, hopefully based on the specific vendor device you are monitoring.

RichanShr
RichanShr

Loglogic in an industry leader in log management. But i am also working for one of those products. May be you could be interested: www.immunesecurity.com

ctp
ctp

I have found what people mean by "Central Log Management" and "Server Monitoring" is often different from what I think it means, so definitions of those things become critical. Recently I had a real problem with communicating my meaning of "Server Monitoring" so I had to write two pages to differentiate between "Event Monitoring" and "Performance Monitoring". Log Managment is similar... do you need logs you can take to court? do you need logs you can cut and paste from? do the logs have to be in a database? Maybe we need a discussion of what these phrases and words really mean? ctp

RichanShr
RichanShr

The logs from all machines should be centrally collected and stored in such a way that it cannot be modified..you know something like a tape backup where you can write only and cannot delete.

ctp
ctp

I find that simply reviewing log files after-the-fact rather than as things happen is not sufficient. Also, limiting my review to only a subset of all possible, will limit my view of what is really happening in my network.

DanLM
DanLM

This is a Unix system(BSD), but I monitor the auth.log, the ftp log, and the firewall log. I've written scripts to perform most of the parsing. This was developed on my home machine. I have just put up a business machine that is web hosting, and my log monitoring will become more diligent. The log parsing routines will be expanded to include the Apache error log and some more of the system logs. I agree, logs should be checked every day. I have what I call ip blocker status report. It notifies me of all ip's that have been blocked, and why. If I see high volume, then I start digging through more of the logs. Also, the network traffic log should be monitored. If you see higher volume(in/out) then what is the norm. This tells you to start looking also. Dan

18th Letter
18th Letter

I need a tool that will collect data that I desire from windows event logs & also acts as a syslog server for my network devices. I then need to be able to print out a report based on a query on any server I am monitoring. GFI Events Manager seems to be alright. What do you think. http://kbase.gfi.com/showarticle.asp?id=KBID002787

ctp
ctp

I was trying to not make a commercial discussion. Try looking at http://www.tditx.com/. The product is ConsoleWorks.

18th Letter
18th Letter

Do you know of any that does everything you mentioned or even comes close?

ctp
ctp

A "perfect" solution would be a single system that collected: - Syslog (from any syslog capable thing not just network devices) - SNMP Traps (and polling) - Serial Consoles - Windows event logs - Windows WMI information You should be able to see individual logs with a common base of date/time. You should be able to have active, immediate scanning of the logs as they occur with notification based on "important" things happening in the incoming log streams. Last of all you should be able to archive the logs and use them without the original collecting application or some specific data base system being present.

Editor's Picks