Notification on Thread Creation
When GDB detects that a new thread is created, it displays a message specifying the thread's identification on the current system. This identification, known as the systag, varies from platform to platform. Here is an example of this notification:
Starting program: /home/user/threads [Thread debugging using libthread_db enabled] [New Thread -151132480 (LWP 4445)] [New Thread -151135312 (LWP 4446)]
Keep in mind that the systag is the operating system's identification for a thread, not GDB's. GDB assigns each thread a unique number that identifies it for debugging purposes.
Getting a List of All Threads in the Application
GDB provides the generic info command to get a wide variety of information about the program being debugged. It is no surprise that a subcommand of info would be info threads. This command prints a list of threads running in the system:
(gdb) info threads 2 Thread -151135312 (LWP 4448) 0x00905f80 in vfprintf () from /lib/tls/libc.so.6 * 1 Thread -151132480 (LWP 4447) main () at threads.c:27
The info threads command displays a table that lists three properties of the threads in the system: the thread number attached to the thread by GDB, the systag value, and the current stack frame for the current thread. The currently active thread is denoted by GDB with the * symbol. The thread number is used in all other commands in GDB.
Setting Thread-Specific Breakpoints
GDB allows users that are debugging multithreaded applications to choose whether or not to set a breakpoint on all threads or on a particular thread. The much like the info command, this capability is enabled via an extended parameter that's specified in the break command. The general form of this instruction is:
break linespec thread threadnum
where linespec is the standard gdb syntax for specifying a breakpoint, and threadnum is the thread number obtained from the info threads command. If the thread threadnum arguments are omitted, the breakpoint applies to all threads in your program. Thread-specific breakpoints can be combined with conditional breakpoints:
(gdb) break buffer.c:33 thread 7 if level > watermark
Note that stopping on a breakpoint stops all threads in your program. Generally speaking this is a desirable effect-it allows a developer to examine the entire state of an application, and the ability to switch the current thread. These are good things.
Developers should keep certain behaviors in mind, however, when using breakpoints from within GDB. The first issue is related to how system calls behave when they are interrupted by the debugger. To illustrate this point, consider a system with two threads. The first thread is in the middle of a system call when the second thread reaches a breakpoint. When the breakpoint is triggered, the system call may return early. The reason-GDB uses signals to manage breakpoints. The signal may cause a system call to return prematurely. To illustrate this point, let's say that thread 1 was executing the system call sleep(30). When the breakpoint in thread 2 is hit, the sleep call will return, regardless of how long the thread has actually slept. To avoid unexpected behavior due to system calls returning prematurely, it is advisable that you check the return values of all system calls and handle this case. In this example, sleep() returns the number of seconds left to sleep. This call can be placed inside of a loop to guarantee that the sleep has occurred for the amount of time specified. This is shown in Listing Eight.
Listing Eight: Proper Error Handling of System Calls.
int sleep_duration = 30;
do
{
sleep_duration = sleep(sleep_duration);
} while (sleep_duration > 0);
The second point to keep in mind is that GDB does not single step all threads in lockstep. Therefore, when single-stepping a line of code in one thread, you may end up executing a lot of code in other threads prior to returning to the thread that you are debugging. If you have breakpoints in other threads, you may suddenly jump to those code sections. On some OSs, GDB supports a scheduler locking mode via the set scheduler-locking command. This allows a developer to specify that the current thread is the only thread that should be allowed to run.
Switching Between Threads
In GDB, the thread command may be used to switch between threads. It takes a single parameter, the thread number returned by the info threads command. Here is an example of the thread command:
gdb) thread 2
[Switching to thread 2 (Thread -151135312 (LWP 4549))]#0 PrintThreads (num=0xf6fddbb0) at threads.c:39
39 {
(gdb) info threads
* 2 Thread -151135312 (LWP 4549) PrintThreads (num=0xf6fddbb0) at threads.c:39
1 Thread -151132480 (LWP 4548) main () at threads.c:27
(gdb)
In this example, the thread command makes thread number 2 the active thread.
Applying a Command to a Group of Threads
The thread command supports a single subcommand apply that can be used to apply a command to one or more threads in the application. The thread numbers can be supplied individually, or the special keyword all may be used to apply the command to all threads in the process, as illustrated in the following example:
gdb) thread apply all bt
Thread 2 (Thread -151135312 (LWP 4549)):
#0 PrintThreads (num=0xf6fddbb0) at threads.c:39
#1 0x00b001d5 in start_thread () from /lib/tls/libpthread.so.0
#2 0x009912da in clone () from /lib/tls/libc.so.6
Thread 1 (Thread -151132480 (LWP 4548)):
#0 main () at threads.c:27
39 {
(gdb)
The GDB backtrace (bt) command is applied to all threads in the system. In this scenario, this command is functionally equivalent to: thread apply 2 1 bt.
Key Points
This article described a number of general purpose debugging techniques for multithreaded applications. To sum up:
- Proper software engineering principles should be followed when writing and developing robust multithreaded applications.
- When trying to isolate a bug in a multithreaded application, it is useful to have a log of the different sequence of events that led up to failure. A trace buffer is a simple mechanism that allows programmers to store this event information.
- Bracket events that are logged in the trace buffer with "before" and "after" messages to determine the order in which the events occurred.
- Running the application in the debugger may alter the timing conditions of your runtime application, masking potential race conditions in your application.
- Tracepoints can be a useful way to log or record the sequence of events as they occur.
- For advanced debugging, consider using the Intel software tools, specifically, the Intel Debugger, the Intel Thread Checker, and the Intel Thread Profiler.
Shameem Akhter is a platform architect at Intel Corporation, focusing on single socket multicore architecture and performance analysis. Jason Roberts is a senior software engineer at Intel, and has worked on a number of different multithreaded software products that span a wide range of applications targeting desktop, handheld, and embedded DSP platforms. This article was excerpted from their book MultiCore Programming. Copyright (c) 2006 Intel Corporation. All rights reserved.
