Debugging applications locally is one of the most difficult tasks of software engineering. Debugging applications remotely is even more problematic. So what do you do when your application crashes at a remote client's site where you don't have a debugger? To address this problem, I developed a straightforward technique that I call "postmortem debugging" to debug program faults independently of where and when they occurred. Key to this technique is the "resymbolizing" of a stack unwind that has been stored to a log file. The technique consists of these steps:
- In the client application, install a handler that's called if the operating system determines a program fault or if an assert fails. This handler walks through the stack and writes the return addresses of all stack frames to a log file. This log file can then be sent via e-mail from the client to the developer.
- At the developer's site, invoke the MapAddr.exe utility program that reads the log file and a map file created when building the client's application. MapAddr.exe then writes a copy of the log file where it adds function names for all addresses by looking them up in the map file.
The result is a snapshot of the call stack (including function names) at the point where the program failed at the client's site; see Figure 1.
****************************************************
*** A Program Fault occurred:
*** Error code C0000005: ACCESS VIOLATION
****************************************************
*** Address: 004011B2
*** Flags: 00000000
****************************************************
*** CallStack:
****************************************************
Fault Occurred At $ADDRESS:004011B2
======== 004011B2 == belongs to ?FunctionC@@YAHHH@Z
+ 66 bytes
with 64 00 00 00 10 00 00 00 2C FF 12 00 02 00 00 00 00 F0 FD 7F
*** 0 called from $ADDRESS:00401223
======== 00401223 == belongs to ?FunctionB@@YAHH@Z
+ 67 bytes
with 64 00 00 00 80 FF 12 00 02 00 00 00 00 F0 FD 7F CC CC CC CC
*** 1 called from $ADDRESS:0040128F
======== 0040128F == belongs to ?FunctionA@@YAHXZ
+ 63 bytes
with 00 00 00 00 02 00 00 00 00 F0 FD 7F CC CC CC CC CC CC CC CC
*** 2 called from $ADDRESS:00401302
======== 00401302 == belongs to _main
+ 66 bytes
with 01 00 00 00 F0 2C 32 00 38 2D 32 00 00 00 00 00 02 00 00 00
*** 3 called from $ADDRESS:0040D04C
======== 0040D04C == belongs to _mainCRTStartup
+ 252 bytes
with 00 00 00 00 02 00 00 00 00 F0 FD 7F F0 0C 23 B2 C8 FF 12 00
Figure 1: Snapshot of a call stack.
To use this approach, all you do is:
- Add a small source file to your application.
- Call a function of this source file at program start that installs the handler using an appropriate OS-function.
- Create and archive map files for all versions of your program. (Every linker can create a map file that contains the addresses of all functions and so on. Although usually disabled, it is an option you must activate. With Visual C++ 6.0, this is done in the "general" category from the Linker tab of the Projects settings.)
There is no need to add a library or install something additional at the client's machine. Everything you need is in the stack of the client machine and in the map file you archived.
Implementation and Sample Code
To illustrate, I present a small program (FaultApp.exe) with three functions calling each other; see Listing One.
Listing One: FaultApp, with three functions calling each other.
int FunctionC(int x, int y)
{
char *p = (char *)0xBADC0DE; // Ummm!!, a dirty bad thing
// Create a access violation
char c = *p;
return 0;
}
int FunctionB(int x)
{
return FunctionC(x, 16);
}
int FunctionA()
{
return FunctionB(100);
}
int main(int argc, char *argv[])
{
InstallWin32FaultHandler();
FunctionA();
return 0;
}
The last function, FunctionC, dereferences a bad pointer, thereby raising an access violation. (Also included are the sources for the fault handler and MapAddr.cpp.)
Although the technique is platform independent, the sample implementation is Windows specific, as it depends on the Win32 SetUnhandledExceptionFilter function. The sample implementation is also specific to Visual C++ 6.0 (although porting to other compilers only consists of the task of parsing its map files). Be aware that the installed handler will not be called if you launch FaultApp.exe using the debugger. In that case, Visual C++ halts debugging. You must launch FaultApp.exe without the debugger to get the installed handler called.
For x86-based UNIX/Posix systems, I provide an equivalent implementation using GCC 3.3 that installs a signal handler using the sigaction function. Interestingly, although the Visual C++ Runtime Library implements the Posix interface (including signal), it does so in a different way than UNIX systems—Windows calls the signal handler after the exit of main, where the stack and registers have already left the state of the fault. MapAddr.cpp also compiles under Linux using GCC 3.3 and can read GCC map files.
Installing the Handler
The Win32 API contains a function SetUnhandledExceptionFilter that makes it possible to install a handler that is called from Windows synchronously when a program fault occurs in your application. Here, "synchronously" means that the handler function is called as if it were a subroutine in your code at the point where the fault occurred. The call stack thereby contains the fault's address as a return address of the installed fault handler. The fault most typically is an access violation or a divide-by-zero problem. The handler is a function that you must provide. Its only parameter is a pointer to a structure holding information about the fault—the type, the code address, and so on. The return code of your handler determines if Windows should continue or abort the program. In my main application, I leave this choice to the client. But in most cases, the resumption of program execution results in more and more faults.
The file Win32FaultHandler.cpp contains the handler Win32FaultHandler (see Listing Two) and a function InstallFaultHandler() that installs this handler (Listing Three).
Listing Two: Win32FaultHandler.
LONG Win32FaultHandler(struct _EXCEPTION_POINTERS * ExInfo)
{ char *FaultTx = "";
switch(ExInfo->ExceptionRecord->ExceptionCode)
{
case EXCEPTION_ACCESS_VIOLATION :
FaultTx = "ACCESS VIOLATION" ; break;
case EXCEPTION_DATATYPE_MISALIGNMENT :
FaultTx = "DATATYPE MISALIGNMENT" ; break;
case EXCEPTION_FLT_DIVIDE_BY_ZERO :
FaultTx = "FLT DIVIDE BY ZERO" ; break;
...
default: FaultTx = "(unknown)"; break;
}
sgLogFile = fopen("Win32Fault.log", "w");
int wsFault = ExInfo->ExceptionRecord->ExceptionCode;
PVOID CodeAdress = ExInfo->ExceptionRecord->ExceptionAddress;
sgLogFile = fopen("Win32Fault.log", "w");
if(sgLogFile != NULL)
{
fprintf(sgLogFile, "****************************************************\n");
fprintf(sgLogFile, "*** A Program Fault occurred:\n");
fprintf(sgLogFile, "*** Error code %08X: %s\n", wsFault, FaultTx);
fprintf(sgLogFile, "****************************************************\n");
fprintf(sgLogFile, "*** Address: %08X\n", (int)CodeAdress);
fprintf(sgLogFile, "*** Flags: %08X\n",
ExInfo->ExceptionRecord->ExceptionFlags);
LogStackFrames(CodeAddress, (char *)ExInfo->ContextRecord->Ebp);
fclose(sgLogFile);
}
/*if(want to continue)
{
ExInfo->ContextRecord->Eip++;
return EXCEPTION_CONTINUE_EXECUTION;
}
*/
return EXCEPTION_EXECUTE_HANDLER;
}
Listing Three: InstallFaultHandler().
void InstallFaultHandler()
{
SetUnhandledExceptionFilter((LPTOP_LEVEL_EXCEPTION_FILTER) Win32FaultHandler);
}
The function Win32FaultHandler() opens a log file, stores the type and code address of the fault, and then calls a function that writes the stack frames to this log file. It returns with the code EXCEPTION_EXECUTE_HANDLER, which handles control the standard handler of Windows (which terminates the application).
I've commented out a section that increments the instruction pointer (in the structure passed to the handler) and then returns with EXCEPTION_CONTINUE_EXECUTION. In the sample FaultApp program, this resumes normal program execution.
