Problems with a rootkit
During this week I've been struggling with a problem with one of our servers.
The server started to collapse with no aparent reason, sometimes we see suspicious processes like this:
./m64 -o stratum+tcp://mine.pool-x.eu:
Searching in google found that it's a miner of bitcoins WTF???
Killing the process and changing the root password seems to have no effect, then I've got to make deeper analysis.
Finally I figured out that the problem was a rootkit installed on the server, then I've taken two curses of action:
1) Remove the rogue software installed
2) Avoid being hacked again addressing the server's vulnerability
1) Remove the malicious software installed by the rootkit
Usually those kind of programs install in temporary folders, look at:
/tmp
/var/tmp
ALERT: Always look with -a flag because usually there are lots of hidden files
ls -la
You must look too for crontabs installed by the rootkit.
cd /var/spool/cron/crontabs
ls -la
Look specially for www-data crontabs or users that you don't control and musn't be there.
Look too in the /etc/passwd file looking for uncontrolled users, use the last command to see last logins into the system.
To find suspicius files and processes you can use this programs, are really easy to use and will give you clues about how to solve the server's vulnerability too:
Chkrootkit: http://www.chkrootkit.org
Rkhunter: http://rkhunter.sourceforge.
Look for the reports generated by those and rm or quarantine the rogue files.
2) Avoid being hacked again addressing the server's vulnerability
Usually those kinds of programs are installed using some webserver vulnerability.
Years ago I've found a similar rootkit that exploited a vulnerability of PHPMyAdmin, the rule of thumb is try to update your software often to avoid this kind of problems.
This time it was a problem with php running in CGI mode.
You must look at clues using chkrootkit and rkhunter, then look at the apache logs for suspicius strings like:
GET /w00tw00t.at.ISC.SANS...
POST //%63%67%69%2D%62%69%...
POST cgi-bin/php-cgi?-d+allow_url_
POST cgi-bin/php5?-d+allow_url_
Often you didn't even need the cgi-bin/php running at all because the standard installation doesn't work through CGI. That's old configuration and probably you don't even need that.
If you can avoid using cgi-bin look at the sites-enabled folder and try to get rid of cgi configurations. Look specially at default site, you can find old lines of config that you don't use at all.
Restart the Apache server after that.
Interesting posts and links:
