You are on page 1of 9

Detecting and Removing Rootkits

What the hell is a Rootkit?

( Definition from ). A rootkit is a collection of tools (programs) that a hacker

uses to mask intrusion and obtain administrator-level access to a computer or computer network.
The intruder installs a rootkit on a computer after first obtaining user-level access, either by
exploiting a known vulnerability or cracking a password. The rootkit then collects userids and
passwords to other machines on the network, thus giving the hacker root or privileged access.

A rootkit may consist of utilities that also: monitor traffic and keystrokes; create a "backdoor" into
the system for the hacker's use; alter log files; attack other machines on the network; and alter
existing system tools to circumvent detection.

If you think/believe your system is Rootkit'ed, read whole of this text before doing anything. If your
system is already possessed, an extra 20 minutes will not change much. We know that the text is a bit long
but we believe it is worth the effort; not because the text is a good one, but because the Rootkits are a
very serious threat and you have to be very careful while removing them.

The procedures described below works for Mandrake distros. We have no experience with other distros
at all. The procedures rely on the rpm tool.

Rootkits are nuisance and can be very serious threats to Linux boxes. Basically they exploit a kernel
weakness (/dev/kmem exploit) to run incognito on a Linux machine, usually sniffing the network
interfaces and recording critical information such as user IDs and passwords.

On a Rootkit infected machine, you will not be able to see any foreign processes by a "ps ax" command
nor see an intruder using a "w" or "who" command. You shouldn't even think about finding anything
useful in your log files.

First things first, you will not be aware of the infection unless you check for it.
A frequently observed effect is that suddenly you will not be able to make ssh connections to your
A sniffer will be running hidden from your "ps" command and logging userIDs and passwords it
comes across.
The program and log files of the sniffer will also be hidden from "ls", "find" commands.
Your computer will have suid programs hidden in various directories.
You will have some "new" accounts on your machine, possibly with userID=0 (root privileges).
Some Linux commands will be replaced with tampered ones which help the rootkit hide its own
files and processes.
Any log files that you might want to see in order to trace the intrusion will be removed.

1 of 9 14-02-2010 22:09
Detecting and Removing Rootkits

As a result, your computer will be totally in the hands of the intruder.

The Google literature mentions a chkrootkit utility. What the chkrootkit utility basically does is it
compares the number of running processes reported by the ps command and the process maintenance
directories under the /proc directory.

We prefer using a manual method which does almost exactly the same, but gives the admin's eyes a
chance to view the running processes and decide whether a suspicious process is running. In the end,
intruders need not use a predefined set of program names for their malicious code. Our technique to see
the actual list of running processes is :

cd /proc
for i in `seq 1 33000`
test -d $i && ls -l $i/exe

You must be root to issue this shell command(s). This shell script

generates a list of integers from 1 to 33000 (you actually need integers from 1 to 32768, but 33000
is easier to type),
for each integer in this list, checks whether a directory with that integer as the name exists (the
kernel manages a directory for each running process using the PID number as the directory name),
if a directory with name "$i" exists, lists the contents of the subdirectory "exe" in that directory.

On a rootkit'ed computer, you will get a listing which looks like (we have clipped the original list):

# for i in `seq 1 33000`; do test -d $i && ls -l $i/exe; done

lrwxrwxrwx 1 root root 0 Jan 4 12:48 1/exe -> /sbin/initsk

ls: cannot read symbolic link 2/exe: No such file or directory
lrwxrwxrwx 1 root root 0 Jan 4 12:48 2/exe
ls: cannot read symbolic link 3/exe: No such file or directory
lrwxrwxrwx 1 root root 0 Jan 4 12:48 3/exe
ls: cannot read symbolic link 4/exe: No such file or directory
. . .
. . .
. . .
lrwxrwxrwx 1 root root 0 Jan 4 12:48 18580/exe ->
lrwxrwxrwx 1 root root 0 Jan 4 12:48 18588/exe ->
lrwxrwxrwx 1 root root 0 Jan 4 12:48 20543/exe -> /var/tmp
lrwxrwxrwx 1 root root 0 Jan 4 12:48 20982/exe ->
lrwxrwxrwx 1 cayfer cayfer 0 Jan 4 12:48 22232/exe ->
lrwxrwxrwx 1 cayfer cayfer 0 Jan 4 12:48 22232/exe ->

2 of 9 14-02-2010 22:09
Detecting and Removing Rootkits

Now, you have to study this long list carefully for any "weird" processes that shouldn't be there. A very
typical sign of infection is the string "(deleted)" on the line which belongs to the init process (or rather
the fake one). If there is the mark "(deleted)" next to some init process, it means a tampered init program
is running rather than the real one. On an infected machine, a tampered init starts during boot, makes
necessary kernel modifications in the RAM to hide the tools the intruder will soon run, then forks the real
init process.

Somewhere down the /proc listing, it is very likely that you will see processes with names like ".pink-sk",
".sniff-sk", ".pink-sniff" etc.etc. It is important that you do not rely on these names only (pink-sk is just
an example); study the list thoroughly. We have seen malicious code running as "/bin/bin", "/usr/libexec

If you find a weird named process, you have to be creative in finding ways to check whether they are part
of the operating system or an application or some malicious code. A good way is to query the RPM
(RedHat Packet Manager) databases for each process you suspect, to see whether they are part of a
package; and then to see whether they are tampered with.

As an example, you can query the RPM database to see whether /usr/sbin/imapd (see above list, last line)
is part of a distribution package (you should type the whole file path):

# rpm -qf /usr/sbin/imapd


In the above example, "rpm -qf" reports that the process running as /usr/sbin/imapd is part of the
imap-2002d-6mdk package.

If you want to check whether /usr/sbin/imapd is a modified copy (potentially malicious code), you can
run the command

# rpm -V imap-2002d-6mdk
S.5....T c /etc/xinetd.d/ipop3

The report produced by "rpm -V" needs to be interpreted carefully.

The first part of the report line ( namely "S.5....T c" ) indicates that

the file /etc/xinetd.d/ipop3 of the package is different in size than the original in the distro package
(the "S" flag),
its MD5 checksum is different (the "5" flag) and
the time stamp has changed (the "T" flag).

Well, we were querying "/usr/sbin/imapd" but it seems that things have rather changed for "/etc/xinetd.d

The catch is that both /usr/bin/imapd and /etc/xinetd.d/ipop3 files belong to the same package and only
/etc/xinet.d/pop3 has been altered so far. For this specific example, changes are not critical; the altered
file is a configuration file and probably the alterations were made by the sysadmin as part of fine tuning
or whatever. Since /usr/sbin/imapd is not reported as a "modified" file, we know that it is safe to have it

3 of 9 14-02-2010 22:09
Detecting and Removing Rootkits

running on our system.

The rpm verification report flags that you might see are:
S : file Size has been changed
M : Mode has changed (such as permissions and file type)
5 : MD5 sum has changed
U : Ownership has changed
G : Group ownership has changed
T : Timestamp has changed
c : Configuration file

(Please see "man rpm" for a few more flags.) Ownership, mode and checksum changes for
binaries are extremely ominous!

You have to go through the process list carefully, paying special attention to the paths of programs and
not ony to names. If you see any program file name(s) starting with a dot, you have probably found it!

The first thing you need to do is to isolate the program file and its log file(s). The problem is that,
although you can see the program file in the /proc listing, you will not be able to see it (them) on the disk
or remove it (them). The modified kernel code in the RAM will hide these from other programs. You will
have to reboot your system in single user mode, avoiding the modified init program to have access to
these files.

The technique is fairly simple, but we must warn you that we have seen rootkits that remove whole
directories of open files during shutdown. We strongly recommend that you do not "restart" or
"shutdown" or "init 0" your system. Probably the malicious "init" program is removing directories
from your system. Instead, do "Alt-Ctrl-Shift-SysRq-U" (unmount disks), then "Alt-Ctrl-Shift-
SysRq-S" (sync) and finally "Alt-Ctrl-Shift-SysRq-B" (boot) to reboot. You will have to exit any X
desktop manager you might be using and return to console mode. (Another good reason to set you system
booting up to console mode rather than to a GUI logon screen). If these keys are not enabled on your
Linux, just enter a couple of "sync" commands and turn the power off. Believe me, the risk is less... You
might as well make backup copies of critical directories like /etc and /var and the homes before resetting
or cycling the power.

While your system is restarting, hit the escape key or the Tab key (depending on whether your LILO is a
gui one or a text screen one) and type in
linux ro init=/bin/bash

to the "boot:" prompt

The command causes the root partition to be mounted read only and the /bin/bash loaded into the
memory instead of init.

Wait till you get the root prompt. Once you get the root prompt, mount the /proc file system by typing

4 of 9 14-02-2010 22:09
Detecting and Removing Rootkits

mount /proc

If this is succesful (we do not know what to do if this step fails) remount the root file system as
read-write using the command

mount -wo remount /

After that, start the devfsd daemon manually by typing

/sbin/devfsd /dev

devfsd will handle /dev links to filesystems on your system.

Now you can either mount all filesystems mentioned in /etc/fstab with

mount -a

or mount only the file systems you need with commands like

mount -t ext3 /dev/hda5 /usr etc. etc.

Your system is now partially up without the malicious init running; therefore the rootkit program files
should now be visible to command like "ls", "rm", "find" etc. For the sample /proc listing above, you
will need to mount the /var directory to see what is in /var/tmp/.pink-sk.

We strongly recommend that you examine each file in the directory where the rootkit is installed. For the
binaries, you can use the

strings binary_file

command to get an idea of what other directories and files the malicious code might be using.

It is also very important to study the rootkit's log files to see what "user name-password" pairs the
instruder has got hold of.

Well, this is the most risky and tedious part. It is not possible to know exactly what damage the instruder
has made to your system. The best and safest way is to install the whole system from scratch. Although it
is relatively easy to build a whole opsys from scratch, the data/site specific applications will force you to
think twice before formatting your disk.

Our prefecerence is to give a try recovering the system. The procedure is:

1. Checking /etc/passwd and /etc/shadow for any alien accounts.

2. Changing the root password.
3. Annulling passwords of EVERY single user account.
4. Removing the rootkit's init, and restoring a clean init program.

5 of 9 14-02-2010 22:09
Detecting and Removing Rootkits

5. Scanning the whole disk space for SUID programs; especially those owned by root.
6. Scanning for modified binary files (our experience indicates that the /sbin/init and Open SSL
server and client binaries are almost always modified, plus some additional SUID programs are
planted in less likely or brand new directories like /dev, /bin/libexec. SUID programs are files
which when executed, runs with the privileges of the file's owner, regardless of the identity of the
user who starts the program. A good example is the "passwd" command. Every user on the
system is allowed to change their own passwords. Changing a password means modifying the
/etc/shadow (sometimes /etc/passwd) file; which is normally not writable by anyone else but root.
The solution is making the '/bin/passwd' file a SUID file and owned by root. When user "jack"
runs the SUID password command, the program will run with the privileges of the owner of the
/bin/passwd file, namely "root", rather than jack's privileges. Rootkits place some SUID files here
and there owned by root, therefore even if you change the root password, they can run the
program with root privileges if they can login to your system with a plain account. That's why you
should annull each and every user password.
7. Recovering modified binaries from distros or by reinstalling them from their tar balls.

First things first; change your root password, change your own password and manually edit /etc/shadow
to annull all other user passwords. To annull an account's password, replace the password field on the
/etc/shadow file with asterixes. Take your time to study the /etc/passwd file to see if there are any weird
accounts, especially with weird home directories.

Rootkits usually keep a copy of the real init program in the /sbin directory. Afterall, they need this to fork
into after their fake init does its work. So far, we have always found the clean init file as /sbin/init-sk.
You might want to make sure that such a file is the real init by issuing a strings command to the binary
and/or comparing it to a clean init file on another computer.

Scanning the system for SUID programs is easy; only if you can trust your "find" program. So, the next
trivial step is to make sure the "find" command is not tampered with.

rpm -qf /bin/find

will report the distro package which contains the "find" command. For Mandrake systems, it is
something like "findutils-4.1.20-1mdk".

To re-install the find program file(s), you have to issue a command like

rpm -Uvh --replacepkgs /your-distro-media/Mandrake/RPMS/findutils-


Having a clean find command, now you can issue the

find / -perm +4000

command to find all SUID programs. You can use "find / -perm +4000 -a -user root" to
find SUID files owned by root (the really dangerous ones).

We are at a point where experience pays... The list of SUID programs can be quite long. Most of the SUID
files you will see are files that really must be SUID. Knowing the real ones from malicious ones requires
some experience and insight. Be paranoid! Double check whether a file should be SUID! Read the

6 of 9 14-02-2010 22:09
Detecting and Removing Rootkits

manual pages, refer to Google, ask more experienced sysadmins, compare with clean systems.

A safer and faster way of checking whether a file should be SUID or not, is comparing its attributes with
the original in the distro package.

Suppose you want to check whether "/usr/bin/chage" should really be a SUID program file. All you need
to do is compare "/usr/bin/chage" with the one in the original distribution. To do this, first you need to
find out what distro package /usr/bin/chage belongs to:

# rpm -qf /usr/bin/chage


Once you know that the RPM package containing "/usr/bin/chage" is "shadow-utils-4.0.3-5mdk", you
need to run the command

# rpm -V shadow-utils-4.0.3-5mdk
..5....T c /etc/login.defs

to see any of the members of the package have been modified. The above example indicates that the
"/etc/login.defs" member of the package has been modified. Since this is a config file, you might decide
that your "chage" command is intact and really should be a SUID file. If it were the other way around,
rpm would report a mode/size change for "usr/bin/chage".

If you come across a file for which you cannot be sure whether it should be a SUID file and be kept on
the system at all, change its owner and group to "nobody" and make a note of this change. If later
something fails, you can reconsider restoring the attributes of this file. To change the owner of a file, you
can use a command like

chown nobody:nobody /usr/bin/somefile

Now it is time to scan the disk(s) for modified binary files...

We strongly recommend that you install the "init", "reboot" and OpenSSH server and client programs
without even bothering testing.

First find the packages containing these programs and then re-install them:

# rpm -qf /sbin/init


# rpm -qf /sbin/reboot


# rpm -qf /usr/bin/ssh


# rpm -qf /usr/sbin/sshd


# rpm -Uvh --replacepkgs /your-distro-media/Mandrake/RPMS/SysVinit-


7 of 9 14-02-2010 22:09
Detecting and Removing Rootkits

# rpm -Uvh --replacepkgs /your-distro-media/Mandrake/RPMS/openssh-clients-

# rpm -Uvh --replacepkgs /your-distro-media/Mandrake/RPMS/openssh-server-

To check ALL your installed packages against the RPM database for integrity, you can use the

# rpm -Va

command. This command will compare all the system programs and applications against their RPM
checksums and report any discrepancies. We strongly recommend that you re-install all packages which
have altered binaries. Altered configuration files and shell scripts are not much of a risk but you might
want to examine any altered scripts and/or configuration files before making a decision though.

We are fully aware that the procedures we have described are not complete or 100% corrective at all. You
should not be shocked if you are rootkit'ed again. We can suggest a simple Perl program to help you catch
the insisting intruder by continuously checking the /proc directory and stopping network services when
an infection is detected. This will at least save the system log files.

while (1) {
@out = `cd /proc; for i in \`seq 1 33000\`; do test -d \$i \&\& ls -l
\$i/exe; done`;

foreach $_ (@out) {
# trap for "pink"rootkit
if ( (/init/ && /deleted/i) || /pink/ || /-sk/ ) {
# test every 15 seconds
sleep 15;

sub stop_net {

# notify sysadm with an email

open(MAIL, "|/usr/sbin/sendmail -t");

print MAIL "From: <user\\n";
print MAIL "Subject: We are stoned!\n";
print MAIL "To: admin\\n\n";
print MAIL qq~
We are rootkit'ted


8 of 9 14-02-2010 22:09
Detecting and Removing Rootkits

close MAIL;
# let the mail to be delivered, 3 seconds should be enough
sleep 3;
# stop the network services
system("/etc/rc.d/init.d/network stop");

If you run the following code in the background, it will detect a new rootkit and bring down the network
interface immediately ( actually in at most 13 seconds) to prevent any further damage.

If you find this document useful, especially if you make use of it to recover a Rootkit'ed system, please
drop us an email to "cayfer at bilkent edu tr" and/or "komur at bilkent edu tr".

Yavuz S. Komur & Can Ugur Ayfer

Jan 2004

9 of 9 14-02-2010 22:09