Linux Memory Forensics
Michael Ford
Forensic analysis is the investigation of an event that involves looking for evidence and interpreting that evidence. In the case of a computer crime in which a system was compromised, the investigator needs to find out who, what, where, when, how, and why.
There are three main areas from which evidence of an intrusion can be gathered. The first and most common is the hard drive. A file system on a hard drive contains the least volatile data. Whether the investigator's strategy involves shutting down the system or just removing the computer's power, the file system will still be there. The investigator's response strategy will dictate what changes are made to the file system. If the file system is shut down or if the investigator issues commands to the system to collect information, the file system may be changed, but in the end, it's still there. There are then many tools, such as The Sleuth Kit or The Coroner's Toolkit (TCT), that can be used to analyze the file system.
The second, and most volatile, of the three areas is network traffic. Once a packet has reached its destination, it's no longer on the wire, and it will only exist briefly in memory on the received system. Regional and national laws dictate the legality of collecting network traffic, but many tools exist to do so. These include intrusion detection systems (IDS), such as Snort, and network monitoring facilities, such as tcpdump or Ethereal.
The third area to investigate is system memory, which I will focus on in this article. The system memory contains everything that is currently happening on the system. If the system is shut down or turned off, this information is lost. We can run many commands before the system is taken down, however, to capture some of this information. For example, we can use date to find the current system time and date. We can use netstat to find a list of the current connections to and from the system and ps to find a list of processes running on the system. But there isn't a command that will show, for example, the changes to files that haven't yet been written to disk. We also cannot be sure that the netstat, date, and ps commands have not been changed by an attacker. An attacker may have modified these and other commands to keep investigators from seeing information about his connections, or he may have replaced them with booby-trapped versions that would erase all of his files, among other things.
Also, running these commands changes their associated access times, which are important when creating a timeline of system activity. Every command that is typed changes the system state. It would be better to save the system memory and analyze it later to find this information. When an attacker breaks into a server and installs a rootkit, many tools can be used to analyze the filesystem of the victim server and find out what the attacker did. Unfortunately, there are far fewer tools and published methods for capturing the memory of an attacked machine and analyzing that image.
One way to save an image of the system memory is to use the crash dump facility. Most Unix operating systems provide a facility for saving the system memory for later analysis. This is useful in cases where a kernel bug causes a crash, and the bug can later be analyzed and fixed. We can use the same facility to analyze the memory after an attack.
I will demonstrate crash analysis to find the answers to three problems. The analysis requires familiarity with the C programming language and some knowledge of Linux kernel structures. To save space, I will omit command output that is irrelevant to the demonstration, and I will replace unneeded structure elements with an ellipsis (...).
The first problem is that, on the victim system, ps does not show all of the processes being run. Originally, ps returned:
[root@host]# ps -f UID PID PPID C STIME TTY TIME CMD root 1211 1209 0 18:15 pts/0 00:00:00 -bash root 1266 1211 0 18:22 pts/0 00:00:00 ./malware 41000 root 1267 1211 0 18:22 pts/0 00:00:00 ./invisible 41001 root 1268 1211 0 18:22 pts/0 00:00:00 ./removed 41002 root 1270 1211 0 18:23 pts/0 00:00:00 ps -fUnfortunately, a trojan ps command was installed to hide any process called "malware". A second problem is that the "adore" Loadable Kernel Module (LKM) was installed. Adore can be used to hide files or processes on the system. Here it was used to make the "invisible" process invisible and to remove the "removed" process from the ps listing forever. So now, ps returns:
[root@host]# ps -f UID PID PPID C STIME TTY TIME CMD root 1211 1209 0 18:15 pts/0 00:00:00 -bash root 1281 1211 0 18:24 pts/0 00:00:00 ps -fThe third problem is that the netstat command has been replaced with a trojan version, and the new one doesn't show all the connections or listening ports on the system. Originally, it showed:
[root@host]# netstat -an | egrep '(^[AP]|ESTAB|LISTEN)' Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 0.0.0.0:41000 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:41001 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:41002 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN tcp 0 0 10.0.7.128:22 10.0.7.1:32831 ESTABLISHEDand now it only shows:
[root@host]# netstat -an | egrep '(^[AP]|ESTAB|LISTEN)' Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTENCreating Crash Dumps with LKCD I will be using the Linux Kernel Crash Dumps (LKCD) package on a Linux 2.4.20 kernel on an X86 system to generate a crash dump. This package consists of three parts: a patch to the kernel, the configuration utility and patches to init scripts, and the lcrash program. To begin, download and install a patch to the kernel to add the source code for the new feature.
# cd /usr/src # wget http://lkcd.sourceforge.net/download/4.1.1/patches/lkcd-2.4.20.diff # cd linux-2.4.20 # patch -p1 < ../lkcd-2.4.20.diffIn the kernel configuration file, the CONFIG_MAGIC_SYSRQ and the new CONFIG_DUMP options must be enabled to allow the crash dump to happen. Also, CONFIG_DUMP_COMPRESS_RLE and CONFIG_DUMP_COMPRESS_GZIP are good options to enable so that the dump is compressed to use less disk space. After building the new kernel and rebooting with it, get the LKCD utilities and install them:
# wget http://lkcd.sourceforge.net/download/4.1.1/lkcdutils/lkcdutils-4.1-1.i386.rpm # rpm -i lkcdutils-4.1-1.i386.rpmYou will also want to review the configuration in the "/etc/sysconfig/dump" file to match your system. Changes are also required to the initialization script specified on the "si" line in /etc/inittab. On Red Hat Linux, this file is "/etc/rc.d/rc.sysinit". The file should be changed to run:
/sbin/lkcd config /sbin/lkcd savebefore the swap partitions are activated to keep the dump file from becoming corrupted as the system is booted. To save a dump file, hit a three-key combination (Alt-SysRq-C), and the system will save a dump file to disk, usually to the swap partition specified by the symbolic link to /dev/vmdump. In some cases, the three-key combination above may not work. In that case, you can put the console in sysrq mode by issuing the following command:
[root@host]# echo 1 > /proc/sys/kernel/sysrqThen on the console, hit the "c" key, which will cause a crash and dump. More information on installing and using LKCD can be found on the LKCD Web site listed in the References section of this article. When the system is rebooted, the crash dump will be saved to a file on disk under the directory specified in the configuration file. During a forensic analysis, this is sub-optimal because it would cause major changes to the evidence. It's better to restore the dump from an image of the evidence disk, rather than from the evidence disk itself. One way to do this is to make an image copy of the swap partition containing the dump and then save the dump from this copy. If we write the copy to an unused partition on the forensics workstation, we can save the dump to a file with lcrash, which is the dump analysis tool that comes with the LKCD package. If we don't write the copy to a partition or other block device, we must modify the lcrash program to be able to read from regular files. In either case, the command to save the dump is:
# lcrash -P -s dumpdev dumpdirwhere dumpdev is the partition or file containing the raw dump, and dumpdir is the directory in which the dump should be saved. The dump will be saved as a filed named "dump.0". Analysis with Crash The "crash" utility is another great tool for analyzing crash dumps, and it understands the dump format used by LKCD. It uses the GNU debugger (gdb), which is provided in the package, for most of its work. I'm using crash version 3.7-5 to analyze the LKCD dump that I got from the victim system. In the common case, the crash command takes two arguments. One is the kernel, which contains debugging symbols, and the other is the dump file generated by the crash dump facility. The debug kernel can be generated by adding the "-g" flag to the "CFLAGS" define in the top-level Makefile when building the Linux kernel. Crash will use this kernel to find the symbols it needs when analyzing a dump file. Make sure that the kernel on the victim system and the debug kernel are the same version and use the same source code. Otherwise, crash will complain that the versions don't match:
# crash vmlinux.dbg dump.0
crash 3.7-5
Copyright (C) 2002, 2003 Red Hat, Inc.
Copyright (C) 1998-2003 Hewlett-Packard Co
Copyright (C) 1999, 2002 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public
License, and you are welcome to change it and/or distribute copies
of it under certain conditions. Enter "help copying" to see the
conditions.
This program has absolutely no warranty. Enter "help warranty"
for details.
GNU gdb Red Hat Linux (5.3post-0.20021129.36rh)
Copyright 2003 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it
under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty"
for details.
This GDB was configured as "i686-pc-linux-gnu"...
KERNEL: /evidence/vmlinux
DUMPFILE: dump.0
CPUS: 1
DATE: Sun Nov 9 18:47:26 2003
UPTIME: 00:33:39
LOAD AVERAGE: 0.00, 0.01, 0.00
TASKS: 36
NODENAME: host
RELEASE: 2.4.20
VERSION: #19 SMP Mon Nov 3 21:47:26 EST 2003
MACHINE: i686 (1799 Mhz)
MEMORY: 160 MB
PANIC: "sysrq"
PID: 0
COMMAND: "swapper"
TASK: c037a000
CPU: 0
STATE: TASK_RUNNING (PANIC)
When crash starts, you'll notice that it provides some interesting information
about the system. Most notable for a forensic investigation is the current date
"Sun Nov 9 18:47:26 2003", the uptime "00:33:39", and the release "2.4.20". The
first two pieces of information will be useful when constructing an event timeline,
and the release will be useful for finding the correct version of the kernel source
code. The panic message "sysrq" also validates that the system dumped because
the investigator instructed it to do so, as opposed to crashing for some other
reason. At the crash prompt, you can use the "help" command to get information
about the other commands available and their usage.
As a first step, we can use the ps command in crash to find a list
of processes that were running on the system:
crash> ps PID PPID CPU TASK ST %MEM VSZ RSS COMM > 0 0 0 c037a000 RU 0.0 0 0 [swapper] 1209 1112 0 c87ee000 IN 1.3 6796 2116 sshd 1211 1209 0 c86d6000 IN 0.9 4312 1412 bash 1266 1211 0 c8580000 IN 0.2 1360 248 malware 1267 1211 0 c8550000 IN 0.2 1360 248 invisible 1312 1211 0 c83ee000 IN 0.9 4668 1460 tcshNote that crash wasn't fooled by two of the hidden processes -- process 1266, which was obscured by the trojan ps, and process 1267, which was hidden by the adore LKM. Adore replaces the "getdents" system call so that when ps looks in "/proc" to get the list of processes, it doesn't find the invisible processes. Crash doesn't use the getdents system call; instead it reads the memory directly. Process 1268, which adore removed from the task list entirely, will be a little trickier to find. In crash, we can enter a kernel symbol name, and the symbol's value will be returned. In the case of a structure, crash will report the values of each element of the structure. The Linux kernel keeps hash lists of all the TCP sockets. The two lists of interest to us are the "established" list, named "__tcp_ehash", and the "listening" list, named "__tcp_listening_hash". These are both members of the "tcp_hashinfo" struct in the Linux 2.4 kernel. We can use these to find any TCP sockets that netstat didn't report:
crash> tcp_hashinfo
tcp_hashinfo = $4 = {
__tcp_ehash = 0xc9f40000,
__tcp_bhash = 0xc9f20000,
__tcp_bhash_size = 16384,
__tcp_ehash_size = 8192,
__tcp_listening_hash = {0xc9e39500, 0xc9e39c00, 0x0, 0x0, 0x0,
0x0, 0x0, 0x0, 0xc86fcb00, 0xc86fc780, 0xc9e67180, 0x0, 0x0,
0x0, 0x0, 0xc9e38400, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc9e39180,
0x0, 0x0, 0xc95e0480, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
...
}
We can use the p command to print out the value of an expression and also
use the * character to de-reference a pointer. When looking through each
of the buckets in the listening hash, we find something interesting when we reach
bucket 10:
crash> p *tcp_hashinfo.__tcp_listening_hash[10]
$13 = {
...
num = 41002,
...
sport = 10912,
...
sleep = 0xc88c06bc,
...
}
This is a socket listening on TCP port 41002. Note that the source port ("sport")
value of 10912 is the network byte order representation of 41002. This can be
shown by swapping the bytes of the short from big-endian byte order to little-endian.
We can find out which process is listening on this port by looking at the task_list
pointed to by "sleep":
crash> p *tcp_hashinfo.__tcp_listening_hash[10].sleep
$16 = {
lock = {
lock = 1
},
task_list = {
next = 0xc8519e80,
prev = 0xc8519e80
}
}
"sleep.task_list" contains a pointer into the stack of the process that is waiting
on this socket. We know by examining the kernel source that each process is represented
in Linux by a task structure that is at the bottom of two pages (8192 bytes) of
memory. By "rounding down" this address by clearing the low-order 13 bits (2^13
= 8192), we get the address of the task structure. We can then have crash cast
this address to a specific type and de-reference it by specifying the type, followed
by the address. Also, simply specifying the type will show a list of elements
in a struct, but specifying the type followed by the "-o" option will also show
the offset of each element:
crash> task_struct c8518000
struct task_struct {
state = 1,
...
next_task = 0xc8548000,
prev_task = 0xc8550000,
...
pid = 1268,
...
uid = 0,
...
comm = "removed\000\000\000\000\000\000\000\000"
...
}
This task is in a "TASK_INTERRUPTIBLE" state. Note that the pointers to the previous
and next tasks in the task list have not been cleared. These could point to other
possibly hidden processes that are worth examining. In this case, one of the processes
is 1267, which we already found, and the other already exited, freeing its memory
for reuse. The pid is listed as 1268, which corresponds to a process we were missing
from our ps listing earlier. We also see that the name of this process is "removed".
Note that the entire comm buffer is printed, even though it contains a NUL-terminated
string, which is why we see the escaped NUL characters. We've now located all
of the processes that were missing.
After examining all of the entries in the listening hash, we can go through
the established hash. The kernel source will show that although the listening
hash is of fixed length, we must look up the length of the established hash
list. This value is stored conveniently in the tcp_hashinfo struct:
crash> p tcp_hashinfo->__tcp_ehash_size $133 = 8192It would take quite a long time to go through 8192 buckets by hand, so we can use the gdb backend to crash and automate this process by setting up a simple loop and printing out the entries that are in use:
crash> gdb set $i=0 crash> gdb while $i < 8192 >if tcp_hashinfo.__tcp_ehash[$i].chain != 0 >p $i >p tcp_hashinfo.__tcp_ehash[$i] >end >set $i=$i+1 >endThis loop uses $i as a variable and increments it from 0 to 8191. If a chain of entries is attached to the hash bucket, the bucket number and the contents of the first entry will be printed out. The following will be printed in this case:
$19 = 296
$20 = {
lock = {
lock = 16777216
},
chain = 0xc9e67500
}
Thus, bucket 296 has something in it. Let's examine it:
crash> p *tcp_hashinfo.__tcp_ehash[296].chain
$5 = {
daddr = 17235978,
rcv_saddr = 2147942410,
dport = 16256,
num = 22,
...
sport = 5632,
...
}
The four values of interest here are the destination address of this connection
"daddr", the source address "rcv_saddr", the destination port "dport", and the
source port "sport". These numbers may not look interesting because they are in
network byte order, and also, the addresses are in decimal, which is difficult
to read. We can use a built-in command to convert the addresses to something more
familiar:
crash> net -n 17235978 10.0.7.1 crash> net -n 2147942410 10.0.7.128We can then use a tool, such as the calculator "bc", to convert the port numbers from big-endian to little-endian. We then find a destination port of 32831 and a source port of 22. We can assume this is the server end of an ssh connection from the host at 10.0.7.1. We can then examine the associated socket structure to find that the connection is in the connected, or established, state:
crash> p *tcp_hashinfo.__tcp_ehash[296].chain.socket
$19 = {
state = SS_CONNECTED,
...
}
We can also find the owner of the socket by looking at the sleep structure:
crash> p *tcp_hashinfo.__tcp_ehash[296].chain.sleep
$65 = {
lock = {
lock = 1
},
task_list = {
next = 0xc840c02c,
prev = 0xc840c02c
}
}
It contains a task_list, which is a pointer to the task_list in a wait_queue_t.
By subtracting the offset of the task_list in the wait_queue_t from the address,
we can see the full wait_queue_t. We can examine kernel header files for the wait_queue_t
declaration and find that this number equals eight:
crash> p *(wait_queue_t *)0xc840c024
$68 = {
flags = 0,
task = 0xc87ee000,
task_list = {
next = 0xc879d8c0,
prev = 0xc879d8c0
}
}
With the task_structure, we can find more information about the process; for example,
we see that its pid is 1209 and its name is "sshd":
crash> p ((struct task_struct *)0xc87ee000)->pid $70 = 1209 crash> p ((struct task_struct *)0xc87ee000)->comm $135 = "sshd\000og\000\000\000\000\000\000\000\000"We can also see a list of sockets used by each process by using the net command. The -s option to net will show us the sockets associated with a specific process. In the following case, we see that the two processes, 1266 and 1267, are listening on ports 41000 and 41001, which were missing from the netstat output:
crash> net -s 1266 PID: 1266 TASK: c8580000 CPU: 0 COMMAND: "malware" FD SOCKET SOCK FAMILY:TYPE SOURCE:PORT DESTINATION:PORT 3 c88c0ea0 c86fcb00 INET:STREAM 0.0.0.0:41000 0.0.0.0:0 crash> net -s 1267 PID: 1267 TASK: c8550000 CPU: 0 COMMAND: "invisible" FD SOCKET SOCK FAMILY:TYPE SOURCE:PORT DESTINATION:PORT 3 c88c0aa0 c86fc780 INET:STREAM 0.0.0.0:41001 0.0.0.0:0By iterating through all of the buckets in the established and listening hash lists, we've now recovered all of the network sockets missing from the netstat output. Saving Memory with memget There are times when the kernel is built without any way to dump its memory to disk. For this, the investigator can use memget. Memget reads the sparse file "/dev/kmem" and transfers the data over a TCP network connection to the address and port specified on the command line for later analysis:
[root@host]# ./memget 10.0.7.1 1026 Saving memory... Saved 45250 pages.Using mempeek to Analyze Saved Memory Once the investigator has transferred the memory image to a forensic workstation, he or she can use mempeek to examine that image. This is a very crude way of examining the memory, especially compared to the interface of crash, but if a dump facility isn't available, it's better than nothing. For example, we know that the system time is stored in the xtime variable in memory. We can find the address of this variable in the System.map file from the /boot directory:
% egrep xtime System.map c03de4b0 B xtimeIt's a good idea to have the kernel source handy during the investigation. We can see that xtime is a "struct timeval" containing the number of seconds and microseconds since January 1st, 1970. Using the peek command at the mempeek prompt returns the requested address and the 32-bit data at that address represented in hexadecimal, decimal, and as a string of four characters:
mempeek> peek c03de4b0 c03de4b0: 0x3faed015 (1068421141) " P.?"We find that the decimal value of xtime.tv_sec is 1068421141. This isn't very useful by itself, but we can use a Perl expression to convert it to a time in the local timezone:
% perl -e 'printf( "%s\n", scalar localtime(1068421141))' Sun Nov 9 18:39:01 2003This matches up pretty closely with the time that crash reported on startup in the previous investigation. The reason for the difference is that memget was run several minutes before the crash dump was taken. We can use mempeek to find the process hidden by adore. Adore sets two bits in the flags element of the task struct to mark a process as invisible. If no other flag bits are set, the value will be hexadecimal 0x03000000. To find the invisible process, we must first find the address of init_task_union.task, which points to the list of processes. We can obtain this address from /boot/System.map on the victim system:
% egrep init_task_union System.map c037a000 D init_task_unionWe can then use mempeek to find the value at that address, as well as other values in the task_struct by specifying hexadecimal offsets after the address. We'll look for the flags, next_task, prev_task, and pid at offsets 0x04, 0x48, 0x4c, and 0x7c, respectively. If we were using crash, we could obtain these offsets by issuing the command task_struct -o. In this case, where the system isn't set up to take a crash dump, it probably doesn't have a debug kernel either so we'll have to determine the offsets by looking at the kernel header files. "next_task" will contain the address of the next process in the doubly linked list of processes, so we can use it to loop through the list:
mempeek> peek c037a000 4 48 4c 7c c037a000: 0x00000000 (0) " " c037a004: 0x00000000 (0) " " c037a048: 0xc9d3c000 (-908869632) " @SI" c037a04c: 0xc83ee000 (-935403520) " '>H" c037a07c: 0x00000000 (0) " " mempeek> peek c9d3c000 4 48 4c 7c c9d3c000: 0x00000001 (1) " " c9d3c004: 0x00000100 (256) " " c9d3c048: 0xc9d30000 (-908918784) " SI" c9d3c04c: 0xc037a000 (-1070096384) " 7@" c9d3c07c: 0x00000001 (1) " " ... mempeek> peek c8580000 4 48 4c 7c c8580000: 0x00000001 (1) " " c8580004: 0x00000000 (0) " " c8580048: 0xc8550000 (-933953536) " UH" c858004c: 0xc86d6000 (-932356096) " 'mH" c858007c: 0x000004f2 (1266) "r " mempeek> peek c8550000 4 48 4c 7c c8550000: 0x00000001 (1) " " c8550004: 0x03000000 (50331648) " " c8550048: 0xc83ee000 (-935403520) " '>H" c855004c: 0xc8580000 (-933756928) " XH" c855007c: 0x000004f3 (1267) "s "The last process has the flags value set to "0x03000000", which corresponds to adore's flags PF_INVISIBLE and PF_AUTH. So, we've now found process 1267, which was missing from the ps listing.
Summary
Sometimes data in memory on compromised systems, such as processes and network connections, are hidden from us by attackers. The attackers may have changed commands to hide this data or may have changed the behavior of the kernel to hide the data from various commands. In this article, I covered two tools that can be used to look at kernel memory. By examining certain kernel structures, I showed how to uncover the hidden processes and network sockets. Crash can be used along with a debug kernel to analyze the memory in a dump file symbolically. In the absence of a crash dump facility, memget can be used to save the system memory, and mempeek can be used to analyze the memory by address. Symbol addresses can be found in the /boot/System.map file on the victim system.
References
adore -- http://wwww.team-teso.net/releases/adore-0.42.tgz
The Coroner's Toolkit -- http://www.fish.com/tct/
crash -- http://freshmeat.net/projects/crash
ethereal -- http://www.ethereal.com/
LKCD -- http://lkcd.sourceforge.net/
memget/mempeek -- http://www.rndsoftware.com/
The Sleuth Kit -- http://www.sleuthkit.org/
snort -- http://www.snort.org/
tcpdump -- http://www.tcpdump.org/
Michael Ford began his career as a network and systems administrator 10 years ago while attending the Massachusetts Institute of Technology. He holds the CISSP and GCFA professional certifications and is the founder of RND Software, Inc., where he performs security research and consulting. Michael can be reached at: [email protected].
