Question

Locked

Strange issue with grep commands eating up all CPU

By deity_chooch ·
Hi all! I have been looking for an answer to this issue for some time and cannot locate a good solution. I have a <acronym title="GNU Not Unix">GNU</acronym>/Linux machine (Fedora Core 2, kernel version 2.6.5-1.35 used as a gateway for wireless customers to connect and get redirected to our signup page. Several months ago, we started experiencing 100% <acronym title="Central Processing Unit">CPU</acronym> usage from this machine and `top` results show multiple `grep` commands as the culprit. These `grep` commands are owned by the `apache` user and, I believe, are being run when a 404 error occurs because of unsubscribed users' attempts to retrieve <acronym title="Web Proxy Autodiscovery Protocol">WPAD</acronym> files (wpad.dat, v8, pki).
The `grep` command itself looks for the <acronym title="Media Access Control">MAC</acronym> address of the user's <acronym title="Network Interface Controller">NIC</acronym> (see commands below). I really hope someone has an answer or at least an idea of where I should look; I'm completed stumped on this one. Thanks in advance!<br/><br/>

<code><br/>
apache 3881 1495 0 13:10 ? 00:00:00 sh -c grep 10.255.236.188 /var/log/dhcpd.log* | grep "DHCPACK on 10.255.236.18
8" | tail -n 1<br/><br/>

apache 3882 3881 4 13:10 ? 00:00:08 grep 10.255.236.188 /var/log/dhcpd.log /var/log/dhcpd.log.1 /var/log/dhcpd.log
.2 /var/log/dhcpd.log.3 /var/log/dhcpd.log.4 /var/log/dhcpd.log.5<br/><br/>

apache 3883 3881 0 13:10 ? 00:00:00 grep DHCPACK on 10.255.236.188

This conversation is currently closed to new comments.

7 total posts (Page 1 of 1)  
| Thread display: Collapse - | Expand +

All Answers

Collapse -

big logs?

by jason_c42 In reply to Strange issue with grep c ...

my first question would be how big the dhcpd logfiles are. If they are sizable, this could definitely be your cause of heartache. I do find it strange that apache does extensive greps on ALL of the dhcpd logfiles instead of just the current logfile.

Another thing you can do is trace the grep processes back to their source by following the trail of parent pids. It may be apache automatically invokes a shell script upon 404 error that scans those files, and you could edit the shell script to take out some of the excessive grep's.

Collapse -

7MB per log, traced to httpd, but cannot find shell script

by deity_chooch In reply to big logs?

First off, thanks for the quick reply. The dhcpd logs are approximately 7MB each and, as you may have seen from my previous output, there are six of them (one current and five rotated). I have traced the `grep` processes back to their perspective `/usr/sbin/httpd` parent processes, but that's where the trail ends. I have yet to find the script which Apache runs, if it exists. Any idea where this sort of script would be placed?

Collapse -

Woohoo! I found it.

by deity_chooch In reply to big logs?

Thanks for your help! I went looking through all the httpd configuration files and discovered that all 404 errors were getting served a non-standard, PHP file that contained a function that runs the commands I see.

Collapse -

another idea

by jason_c42 In reply to Woohoo! I found it.

I was thinking... if you preferred not to alter the apache configuration, you could also rotate the dhcpd logs more frequently.
I'm guessing you have a cron routine setup to rotate them, if you made it run more frequently that should cut down on their size. The drawback there is that you wouldn't keep logs for a long.
If that is a concern, modifying the log rotation routine would be another option. You could create a new directory, "archive" for example, and move rotated logs into it so apache couldn't find them.

Collapse -

Got Rootkit?

by robo_dev In reply to Strange issue with grep c ...

There's the Random JavaScript Toolkit which hits Apache pretty hard.

http://blog.cpanel.net/?p=31

http://servertune.com/kbase/entry/258/

Collapse -

Thanks for the information!

by deity_chooch In reply to Got Rootkit?

Thanks for the information, I will definitely look into this.

Collapse -

I would start with your auth.log

by robo_dev In reply to Thanks for the informatio ...

there have been some SSH hacks that are pretty horrible, and very very stealthy.

rootkit hunter
http://www.skullbox.net/rkhunter.php

Back to Networks Forum
7 total posts (Page 1 of 1)  

Related Discussions

Related Forums