What Are MACtimes?

Table 1 lists the various times and their meanings. The operative word here is "last" -- MACtimes only keep track of the final time a file is disturbed; historical data is impossible to uncover.

Reading a file generally changes the atime attribute, although NTFS unfortunately updates the atime only if the updated atime would be an hour or more later than the previous atime. Note also that when a program is run, the atime of the executable file changes because, obviously, the contents of the executable file must be read before execution. Many systems also let you either totally disable atime updates or mount individual file systems with the atime updates turned off (primarily for performance reasons). This can be useful when examining evidence.

Mtimes are changed by modifying a file's contents. Ctime keeps track of when the metainformation about the file has changed -- the owner, group, file permission, and so on. In systems without dtime, ctime may be used as an approximation of when the file was deleted, although it really points to the time when the last link that pointed to the file was removed.

Oddly enough under NTFS, when a file is copied, the mtime of the target file is the same as the original file, while the atimes and ctimes behave as expected. This can make a file appear as though it was created after it was modified.

UNIX system directories store only the names of the files and their corresponding inode numbers. The rest of the information about the file, including the location of data blocks and many other attributes (such as the file permissions, owner, and the like) are kept in the actual inode of a file. NTFS doesn't use the directory to store much information either; instead, it relies on a Master File Table (MFT) that contains information about all the files in an NTFS volume.

So far, so good, you may be saying -- but so what? Let's examine an example of using MACtimes.

Compiling Information About Activity

In the examples that follow, we'll be using two programs included in The Coroner's Toolkit (available at http://www.fish.com/ forensics/ and http://www.porcupine.org/ forensics/). "Grave-robber" is a program that captures all kinds of data from a system, usually from the most volatile to the most stable in an effort to preserve the more ephemeral data that mere data collection can destroy or lose the chance to grab -- program memory, network connections, and yes, MACtimes, among other data types. The "mactime" program is a way of selectively viewing a slice of historical time from the database; last night from 10:00 pm to 3:00 am, for instance. It generally uses the database that "Grave-robber" previously generated.

One of the simplest security-oriented things to conceptualize is a login from a remote host to a target system. What actually happens? Under UNIX, in basic terms, the inetd program listens to the telnet port and forks off the telnet daemon. Telnetd talks the telnet protocol, then executes the login program. Login authenticates the user and becomes the shell after updating the login accounting files. Running the "mactime" tool after such a login reveals this in a bit more detail; see Figure 1.

The "mactime" program output illustrates this neatly, displaying all the programs executed (inetd, in.telnetd, login, and csh), configuration and authentication files used (passwd, group, ttytab, and motd), and finally, the system accounting files (utmp, wtmp, and lastlog) -- the only files that were actually modified in this sequence.

In a more general sense, it is surprising how little actual activity happens on a typical workstation or desktop system. While we often think of systems as being very busy, constantly running programs in the background and foreground, even servers generally run the same set of programs over and over, either reading or writing to the same set of files.

If you're hunting for signs of malicious or anomalous activity, this is an ideal situation, because most problems involve something a bit out of the ordinary -- at least, I hope they do!

For example, assume that it's just after dawn on August 7th, 2000, when you observe the output in Figure 2 of the "mactime" tool on your Linux system (compared to other activity, this will generally stick out like a sore thumb).

What has happened? The cc, cpp, as, and ld commands were all run (or at least read; you may assume from the context that they were actually executed), and several header files (including lastlog.h and utmp.h, which are fairly unusual header files that describe the format of system login accounting files) were all accessed within a second or two. It seems fairly obvious that a C program was compiled at approximately 15 minutes past midnight the night before.

Combine this with the fact that your system doesn't show any login activity of any sort last night, and it becomes a relatively easy leap for a digital detective to deduce that someone probably broke into your system and compiled a cloak or stealth program that removed their presence from the system accounting files. Further examination of the system for nefarious activity might well be a good idea.

Patterns such as these abound in systems, and many activities will generate reasonably clear indicators of actions (unless the system is tampered with). Of course, the patterns may be obscured by the normal use of a system; a developer, for instance, might be running the compiler fairly often and wouldn't necessarily see what might seem like another random header file being accessed. Even in such cases, however, there are files that are almost never used (for example, RPC header files), programs that are almost never run (especially note those that execute with system privileges), and important files that are rarely changed or shouldn't exist (password files, files that grant remote access to users, and so on). In all such cases, MACtimes can be very useful, especially when combined with other methods of data collection and analysis.

Caution: Crime Scene

Part of the usefulness of MACtimes is not only can you get them from a running machine, but you can also get them from a dead disk (for instance, by mounting a disk from a system computer that was given to you). In addition, your machine doesn't have to be the same operating system type as the computer that generated the data; my Red Hat Linux system can read and collect MACtimes from an NTFS file system, for example.

Collecting MACtimes must be done cautiously, however, as they are extremely ephemeral -- a stiff electronic breeze can destroy any hope of recovering them. While lstat()ing a file does not change the MACtimes, opening a directory for reading will change the atime, so you must be certain to lstat() directories before opening them and examining their contents. Digital signatures (a mathematical method to sign or stamp data with a mark that can be easily validated but is difficult computationally to forge) are a useful and commonly used tool for a variety of forensic or administrative purposes, but must be done after the lstat() -- because reading a file changes the atime of that file.

If doing a serious investigation you'll ideally want to work from a duplicate of the media rather than using the original data. Failing that, mount the media as read-only or, at the very least, turn off atime updates so that you don't inadvertently destroy or alter the data and come up with incorrect conclusions.

System Activity

Computers are often thought of as fairly impenetrable black boxes, and the paucity of available and authoritative documentation by vendors only adds to this mental model. And while what a system does from the time the power switch is turned on to the time you're able to log on is roughly understood, to really understand a system -- especially if tracking down problems -- more information is often required.

MACtimes can provide quite a bit of valuable general information about what programs and files are used on operating system startup or shutdown. The easiest way to access this data is by running a "mactime" program immediately after the system is started (or putting it into the system bootup process -- if you're aware of what the last step is, that is!). While space precludes listing (let alone analyzing) all the files, Table 2 lists some popular operating systems I run, and a rough number of files they use for booting up, as reported by "mactime." As you can see, even the lowly DOS/FAT file system roughly supports MACtimes; it keeps certain times along with the filename and other information in the directory entry itself, but has various significant problems and differences that prevent a full discussion alongside NTFS and UNIX filesystems in this space. NT 4's low numbers can be directly traced to the system not running any servers such as Exchange, Back Office, and Internet Information Server (IIS), unlike the UNIX systems. Also interestingly, given the various systems' ways of updating atimes, NTFS generally shows you the time when a program or file was first used in the bootup sequence, while UNIX systems show you the last time it was disturbed. MacOS X Developer Preview 4 is Apple's prerelease Mac/FreeBSD-based OS, which also runs relatively few network services.

What does this tell me? Taking the numbers with a grain of salt -- it does seems reasonable that as the number of files (programs or configuration files, primarily) used by a system grows, the complexity of the system would also generally climb. And complexity is one of the main enemies of security -- it causes programming errors, design miscalculations, and other computer faux pas. Knowing nothing else, if given two similar systems where one system uses twice as many files as the other, one would expect the less complex system to be significantly less prone to security flaws.

On a more tangible level, these numbers show what you're up against to understand a modern operating system. Simply turning on the power causes the system to use at least a hundred and possibly more than a thousand programs, configuration and data files, and the like. And that's just the start. When you throw in applications -- some that are significantly larger than the kernel of the operating system they run on (editors, windowing systems, and browsers come to mind) -- you're up against a sizable number and a real problem. Most of the files just mentioned can subvert a system absolutely, bring it to its knees, or even prevent it from functioning. How can you possibly begin to analyze or uncover evidence of clever corruption if you don't know what's going on?

Finally, the rationale for removing windowing systems from firewalls and other security-sensitive systems becomes blatantly obvious, as it can be seen that they add tremendously to the overall system complexity. Operating systems that cannot operate without a windowing system have an inherent security disadvantage and should, in general, be eschewed over those that can run without.

As unbelievable as it might seem, MACtimes can even be used to help secure a system. Virtually all operating-system vendors ship a plethora of features, programs, and files that could compromise system security and are rarely or never used. Ideally, you would like to remove such items to keep the system, as they say, lean and mean (or, as we'd like to say, smaller and less complex). DDJ readers, in particular, are especially aware that many bugs and problems are caused by seldom-used code and features.

A security conscious administrator could create a list of all files on a system and compare them to the list of files that have been accessed since the computer was first turned on (or, for long-running computers, for the last year or so). Of course, the more time the computer has been in use the better -- you don't want to try this the same day you install the system! But if a program -- especially one that runs with system privileges or talks to the network -- hasn't been used in six months to a year, then it's probably a good candidate for removal. (Beware of systems that run for long times without crashing. UNIX systems can have uptimes of a year or more. If this is the case, you could accidentally delete important files necessary to start the computer. In other words, use common sense!) Obviously you should be judicious with the rm or del command. Some systems do look for the presence of files even if they never use them. But despite all the caveats, especially on UNIX systems with their armada of SUID and SGID programs, removing unused programs can considerably improve the security of a system.

Life after Death

One might reasonably expect that after a file or directory is gone, the MACtime has little use -- however, this is not always the case. In UNIX systems, when a file is removed, the ctime is set to the time when the last link to the file has been destroyed, which is most often at the time it was deleted (if your system supports the dtime file attribute you can use this instead, of course). The inode is also deleted from the directory entry, making recovery of the data or its associated name difficult (but not impossible, as we shall discuss in another column). While some file attributes are erased, at least the ownership and MACtimes are preserved in the UNIX flavors I examined, and if you can get to the system before it syncs, you can recover even more information. NTFS also doesn't remove all the file information when a file is deleted; instead, it sets a flag in the file record of the MFT telling the file system that the file is not in use anymore.

MACtimes of inodes that were once attached to files may be recovered by using the "ils" tool (also found in The Coroner's Toolkit).

As an example of the use of MACtimes of removed files, I was asked to examine a system that had been broken into and had a significant portion of the file system removed by a malicious intruder. After recovering the inodes and the corresponding MACtimes from the unallocated file system, it was simple to uncover the (significant!) traces of all the recently deleted files by using the "mactime" tool. By then comparing the MACtimes of the live file system with those of the unallocated file system (combined with the ownership information recovered from the inodes), I was able to not only pinpoint when the intruder started the damage, but I found out where they started destroying files, what account was used when the intruder logged in, as well as what time the miscreant entered and left the system.

Problems with MACtimes

Like every other tool, MACtimes do not provide the proverbial silver bullet solution. Their most obvious shortcoming is that they only report on the last time a file has been disturbed and, hence, have no way of reporting on the historical activity of a file or directory. A program could run a thousand times and you'd only see evidence of a single occurrence.

MACtimes also degrade over time, displaying a sort of digital Alzheimer's. As the activity goes further back in the past, your odds of fighting a losing battle increase. They're also less useful on busy multiuser systems with lots of activity -- or on systems that exhibit behavior that emulates the actions of intruders or other problems you wish to investigate.

In addition, UNIX systems have the touch command, which can change atimes and mtimes. Both NTFS and UNIX filesystems can also use the utime() system call to change those two times, as the Perl code in Example 1(b) demonstrates.

Ctimes are more difficult to change in time on UNIX systems (NT provides the SetFileTime() system call, which can be used to change all three times at once), but if an intruder has privileged user access, he can reset the system clock and then change the ctime, or alternately write directly to the inode. Changing the system clock can cause other warning flags to sound; however, most systems don't like time going backwards or hopping around, and log files or other signs might tip off such activity.

Certainly, do not use MACtimes as a replacement or alternative for digital signatures or other forms of file validation! Simply because the times on, say, your "ping" binary indicate that it hasn't been used in a while doesn't mean that the intruder didn't put a Trojan horse on it and then simply modify the access times.

As a final caveat, NTFS has an interesting way of dealing with daylight savings time when using the Event Viewer or user-level methods of examining a file's time information. When daylight savings arrives, NT changes the reported times on files and event logs retroactively by an hour, even if the files were created before the time change! This can make precise accounting of events difficult.

Conclusion

I like to think of MACtimes as a veritable Ginsu knife for security and analysis -- a tool or concept that can be used to slice, dice, understand, and examine a wide variety of activities and behaviors. However, to utilize MACtimes to their highest potential, you should combine them with other types of informational gathering methods, such as program profiling, process accounting, system and program logs, kernel auditing, and auditing in general. Collect MACtimes as quickly as possible, and before gathering any other forensic data that might destroy them, and perhaps paradoxically, don't completely trust them -- at least, don't base your decisions solely on MACtimes, irrespective of how damning their evidence might be.

And while NTFS is hampered in that atimes are not always updated when a file is accessed (Microsoft, please change this, or at least make it an option for administrators!), MACtimes are useful on systems that utilize the NTFS or UNIX filesystems (and, although mostly beyond the scope of this piece, Microsoft's FAT file systems, the Mac HSF+ file system, and others).

DDJ