Scanning Linux For Intrusion With RKHunter

RKHunter (or Root Kit Hunter) is a program that can be used to scan a Linux machine to see if there is anything there that might be a sign of a security breach. It will scan all of the files on the system and look out for any suspicious files or unexpected changes to system files that might indicate a security breach. Just like anti-virus systems it has a database of root kit definitions that it will use to compare files against to see if they are infected but will also just check for changes to core system files.

As the tool will also compare the current status of the system with the known good states it is really important to install RKHunter when you first set up the system. RKHunter also integrates with tools like yum and apt so it will know when you have installed any officially sanctioned programs and won't warn you about them when it next scans your system. This reduces the chance of false positives being generated.

To install RKHunter on Centos/Redhat you can use yum.

sudo yum install rkhunter

To install RKHunter on Ubuntu you can use apt.

sudo apt-get install rkhunter

To check for latest version of program just run it with the --versioncheck flag. You always need to run RKHunter as sudo so the examples on this page will always include that.

sudo rkhunter --versioncheck

As RKHunter will compare your system files with a database of root kit definitions it's important to make sure that this definitions list is up to date. Running RKHunter with the --update flag will update all of the root kit definitions available.

sudo rkhunter --update

RKHunter doesn't just check for the presence of root kits, it will also look over system files and check to make sure that they haven't been changed. A common approach on a hacked machine is to swap out core system programs like ls or cd for malicious versions so checking for these sort of changes is a really quick way of seeing if your system has been breached. Of course, you first need to build up a database of your system files, which can be done with the --propupd flag.

sudo rkhunter --propupd

You can now run a manual check with the --check flag, this will run through all of the checks that RKHunter does and tell you what passed and what produced a warning. Running this with the --skip-keypress flag will prevent RKHunter from asking for you to press the enter every time it finished a test batch.

sudo rkhunter --check --skip-keypress

Running the --check command on it's own gives you pages and pages of output with the things to look out for buried within that output. To show only the warnings then you can include the --report-warnings-only flag. This allows you to check for the important things.

sudo rkhunter --check --report-warnings-only --skip-keypress

When RKHunter has run it will write a log to the following location on most systems. The logfile will contain a lot more information than the output of the program and will show you all of the warnings found during the scan. This logfile will typically be found at the following location.

/var/log/rkhunter/rkhunter.log

Obviously, logging into your system every day to run RKHunter will get tiring. The best approach to this is to run the program via cron. There are a few ways that RKHunter can be run via cron, but the best way is generally to run RKHunter directly from the cron task with the --cronjob flag. If this option is used no output is produced and any warnings found will be automatically emailed to the root user. The --versioncheck flag is used to make RKHunter check that it is up to date and to create a warning if an update is available. We also want to get RKHunter to update its rootkit definitions database, which is done with the --update flag. When running the program via cron it is also necessary to tell it what output to be shown. This is normally done using the --rwo flag, which stands for 'report warnings only'. Here is a typical crontab entry that will run RKHunter every day at 5:30am.

30 5 * * * /usr/local/bin/rkhunter --versioncheck --cronjob --update --rwo

An added bonus of using the --cronjob flag is that RKHunter will email the admin user to tell them of any warnings or problems.

Once you have updated your rootkit database and run your first scan you can take a look a the configuration file to see if anything needs altering. There are a few options available here but perhaps the most useful is the MAIL-ON-WARNING setting. This is used by RKHunter to send an email to an administrator when warnings are found during a unmanned scan of the system. The RKHunter configuration file can be found at the location /etc/rkhunter.conf on most system. The following shows an email address added to the MAIL-ON-WARNING setting.

MAIL-ON-WARNING="[email protected]"

If you want to check that you haven't broken anything in your RKHunter config then you can run the command with the -C flag. This will run a syntax check over the config files and warn you of any potential problems.

sudo rkhunter -C

I mentioned at the start of this post that RKHunter integrates with apt and yum. This needs to be done via the APT_AUTOGEN flag set in the RKHunter setup file at /etc/default/rkhunter (not the main config file at rkhunter.conf). If this line isn't present then add it in. Now, when you update or install a program via apt or yum, RKHunter will automatically register this change instead of flagging it as a warning.

APT_AUTOGEN="true"

Of course, what you do after a warning has been issued is another matter, RKHunter is only designed to be a warning system and can only inform you of potential problems it finds. I should point out that when running the tool for the first time on a new system you will see plenty of warnings appear, this is why it's important to run the tool when you first install it. If you receive an email stating that a warning has been found during normal operation then you should review the warnings and potentially login and see what is going on. You might need to read the logs see what the warnings contained an inspect the files in question. Some warnings are pretty benign and will turn out to be unexpected updates to some system files, but you should never ignore these warnings and assume everything is ok. If they turn out to be evidence of a compromise then you need to take steps to fix the problem and then tidy up any mess.

Personally, if you do find evidence of a breach I wouldn't bother trying to put the server back into order. In my opinion, the best and most effective way to recover from an attack is to delete the server and start again. If you have provisioned your servers correctly then you should be up and running in a few minutes, rather than the hours it might take to put the server back together again. Even if you cleaned things up there is no guarantee that everything will be clean as backdoors may have been engineered to allow easy future access. That said, the first thing you need to do before you wipe the server is figure out how the breach happened. This can provide useful information for preventing future attacks.