Discussion Locked

Log File Analysis

0 helpful
2 replies
Closed to new repliesThis discussion is closed. It stays readable, but it takes no new replies.

I am working in an environment where I have a number of servers that are creating a large number of non-similarly format log files, not Webserver log files.

I want to be able to search through all these different log files on the different servers for particular phrases, typically error messages, warnings etc.

Does anyone know of any programs they can recommend for this operation.

I am thinking of something that would allow me to specify numerous search strings and paths and it would need to be fast in operation as there are hundreds of Meg's of log files and be flexible in the reporting section.

Any feedback would be most appreciated. Thanks.

2 replies

by bhannigan

use a data transformation tool to convert each of the files to a normalized format. load that into a database and report at will using off the shelf tools. you didn't mention architecture, but tools of this nature are available in most of the popular ones.

by andy carroll

Thanks bhannigan for that.

The system set up is: the Network OS is NT 4, and I have a large bespoke software system writing numerous log files in raw ascii to temp directories on 10 different servers. When I originally posted the first request for assistance the logfiles were 4+ Gigabytes, currently these are 10+ Gigabytes. We do currently have a large amount of debug information being logged and this will reduce dramatically when we are happy with the performance of the system and the developers turn the debuging info down / off, but logs will still be fairly large.

I don't currently have any database skills but it looks like I am going to have to learn some pretty quick.

I need to be able to monitor these logs in as close to real time as possible. I will probably only need to monitor a smaller number of these logs closely eventually, but it will take sometime to decide which logs need this level of monitoring and in the meantime we need to monitor everything. Any further advice or recommendations for particular tools or procedures would be most appreciated.

This discussion is closed to new replies.