Networks

Ask something
Question Locked

Strange issue with grep commands eating up all CPU

0 helpful
8 replies

by deity_chooch

Closed to new repliesThis discussion is closed. It stays readable, but it takes no new replies.

Hi all! I have been looking for an answer to this issue for some time and cannot locate a good solution. I have a GNU/Linux machine (Fedora Core 2, kernel version 2.6.5-1.358) used as a gateway for wireless customers to connect and get redirected to our signup page. Several months ago, we started experiencing 100% CPU usage from this machine and `top` results show multiple `grep` commands as the culprit. These `grep` commands are owned by the `apache` user and, I believe, are being run when a 404 error occurs because of unsubscribed users' attempts to retrieve WPAD files (wpad.dat, v8, pki).
The `grep` command itself looks for the MAC address of the user's NIC (see commands below). I really hope someone has an answer or at least an idea of where I should look; I'm completed stumped on this one. Thanks in advance!

<code>
apache 3881 1495 0 13:10 ? 00:00:00 sh -c grep 10.255.236.188 /var/log/dhcpd.log* | grep "DHCPACK on 10.255.236.18
8" | tail -n 1

apache 3882 3881 4 13:10 ? 00:00:08 grep 10.255.236.188 /var/log/dhcpd.log /var/log/dhcpd.log.1 /var/log/dhcpd.log
.2 /var/log/dhcpd.log.3 /var/log/dhcpd.log.4 /var/log/dhcpd.log.5

apache 3883 3881 0 13:10 ? 00:00:00 grep DHCPACK on 10.255.236.188

8 replies

by jason_c42

my first question would be how big the dhcpd logfiles are. If they are sizable, this could definitely be your cause of heartache. I do find it strange that apache does extensive greps on ALL of the dhcpd logfiles instead of just the current logfile.

Another thing you can do is trace the grep processes back to their source by following the trail of parent pids. It may be apache automatically invokes a shell script upon 404 error that scans those files, and you could edit the shell script to take out some of the excessive grep's.

by deity_chooch

First off, thanks for the quick reply. The dhcpd logs are approximately 7MB each and, as you may have seen from my previous output, there are six of them (one current and five rotated). I have traced the `grep` processes back to their perspective `/usr/sbin/httpd` parent processes, but that's where the trail ends. I have yet to find the script which Apache runs, if it exists. Any idea where this sort of script would be placed?

by deity_chooch

Thanks for your help! I went looking through all the httpd configuration files and discovered that all 404 errors were getting served a non-standard, PHP file that contained a function that runs the commands I see.

by jason_c42

I was thinking... if you preferred not to alter the apache configuration, you could also rotate the dhcpd logs more frequently.
I'm guessing you have a cron routine setup to rotate them, if you made it run more frequently that should cut down on their size. The drawback there is that you wouldn't keep logs for a long.
If that is a concern, modifying the log rotation routine would be another option. You could create a new directory, "archive" for example, and move rotated logs into it so apache couldn't find them.

by deity_chooch

Thanks for the information, I will definitely look into this.

This discussion is closed to new replies.