Avoiding Trouble with Mouse Capture
Chris Branch
Windows normally sends mouse messages (WM_MOUSEMOVE, WM_LBUTTONDOWN, etc.) to whichever window is beneath the mouse cursor. However, a variety of situations, such as implementing most kinds of drag-and-drop, require you to arrange for a single window to receive all mouse input, even when the mouse goes outside the bounds of that window. The standard Windows functions SetCapture() and ReleaseCapture() allow a window to “capture” the mouse (claim all future mouse input for its own) and “release” the mouse (restore the default behavior for mouse input).
I recently spent time fixing some of my existing code to eliminate several problems I had with mouse capture. In this article, I’ll describe what went wrong, and how to avoid the same problems in your code.
Mouse Capture
Sending mouse input to whichever window is beneath the mouse cursor is a good default behavior, since it basically lets the user explicitly deliver input to any window visible on the screen, and you can divide the labor of handling input logically by window. Sometimes, though, the default behavior is not appropriate, and you’ll want a specific window to receive all the mouse input, even when the mouse cursor is over other windows. Many drag-and-drop operations fall under this category.
Figure 1 shows a code fragment to implement some kind of drag-and-drop operation. Typically, a drag operation would begin when you detect a WM_LBUTTONDOWN message in a given window. You would then call SetCapture() to ensure your window procedure received all the mouse messages to follow, even after the mouse moved outside the window. For each WM_MOUSEMOVE after the initial WM_LBUTTONDOWN, you might draw an image that represented whatever is being dragged. Finally, upon receiving a WM_LBUTTONUP, you would note the location where the item is being dropped, perform any appropriate action, and call ReleaseCapture() so that normal processing of mouse input messages would resume.
The Problem
In a normal scenario, the code shown in Figure 1 will work just fine. The problem occurs if the system decides to cancel the mouse capture in the middle of the capture operation. If this occurs, the WM_LBUTTONUP message may never be received and the window never gets an opportunity to clean up. In the code fragment from Figure 1, the result is a memory leak. Also, the bDragging flag never gets reset, so the window is “stuck” in dragging mode.
Under Windows 95 and Windows NT 4.0, I’ve seen several situations where the system will automatically cancel a mouse capture that is currently in progress. For testing purposes, the easiest to reproduce is pressing Alt-Tab to switch to a different application in the middle of a capture operation. Windows will cancel the mouse capture and switch to the new application. Pressing Ctrl-Esc to display the “Start” menu also has the same effect.
A scenario that is more realistic but harder to reproduce occurs when a background application calls SetForegroundWindow(). I’ve seen several applications (most notably Internet Explorer) use this technique to get the user’s attention when some important event has occurred. If a background application calls SetForegroundWindow() while the mouse is captured, the result is the same as the Alt-Tab example I described earlier.
The bottom line is that the system can cancel your window’s mouse capture at any time. To handle this gracefully, you must detect and handle cancellation of mouse capture.
The Solution
There are several messages that can help solve this problem. Just before the system “steals” the mouse capture from your window, your window will receive a WM_CANCELMODE message. By default, DefWindowProc() calls ReleaseCapture() when it receives a WM_CANCELMODE message. Thus, if you pass this message on to DefWindowProc() without handling it yourself, your window will automatically lose mouse capture.
For my first attempt at fixing this, I processed WM_CANCELMODE and simply returned TRUE instead of passing the message to DefWindowProc(). I reasoned that this would prevent the system from calling ReleaseCapture(), so mouse capture should remain in effect. Based on the output from the Spy++ program that ships with Visual C++, my window was able to retain mouse capture after another application called SetForegroundWindow(). However, the other application was now covering my window so I was unable to see what I was dragging!
Clearly, a better response to WM_CANCELMODE is to cancel the drag operation, perform any required cleanup, then pass the message on to DefWindowProc(), which will release the mouse capture. This is the solution I implemented in my own code. The revised code fragment is shown in Figure 2.
The code in Figure 2 will work correctly under Windows 3.1 as well as Windows 4.0 (Windows 95 and Windows NT 4.0). However, if your application is targeted for Windows 4.0 or above, there is another solution which is actually the method recommended by Microsoft.
The WM_CAPTURECHANGED message is new for Windows 4.0 and is sent anytime ReleaseCapture() is called. The system always sends WM_CAPTURECHANGED regardless of whether the application explicitly calls ReleaseCapture() or the system calls ReleaseCapture() when processing WM_CANCELMODE. The advantage is that your cleanup code can be located in one place. The revised code using WM_CAPTURECHANGED is shown in Figure 3.
Problems with WM_CAPTURECHANGED
There is one potential “gotcha” regarding WM_CAPTURECHANGED. Since WM_CAPTURECHANGED is new for Windows 4.0, the system will not send this message to any window created by an application whose “expected windows version” is less than 4.0. The expected windows version is embedded in the executable file (.exe or .dll) at link time and can be viewed using a file dumping utility such as Borland’s TDUMP.
When you call CreateWindow(), you pass an hInstance parameter which associates the window with your application. Internally, CreateWindow() passes that hInstance to the undocumented KERNEL function GetExpWinVer(). GetExpWinVer() determines which module (EXE or DLL) the hInstance refers to and returns the expected Windows version number that was embedded in the module’s executable file header by the linker. If this version number is 4.0 or higher, a special flag is set in HWND’s internal data structure. When ReleaseCapture() is called, either explicitly by the application or implicitly via DefWindowProc(), Windows checks this special flag. If the flag is set, Windows sends a WM_CAPTURECHANGED message. Note that this is not a 16- versus 32-bit issue — it’s a version number issue.
This has some interesting implications. For example, suppose you have a DLL that implements a custom control that can be used by multiple client applications. Because each HWND is effectively version stamped during creation, the window procedure inside your DLL may or may not receive certain messages (e.g., WM_CAPTURECHANGED) depending on who called CreateWindow() to create the control. It doesn’t matter that the DLL which implements the control is marked for Windows 4.0 and fully understands (and expects) these new messages. The lesson here is that you can’t rely on WM_CAPTURECHANGED unless you’re dealing exclusively with code targeted for Windows 4.0 or higher.
A related issue is that it’s easy to accidentally end up with a Win32 module that is not marked for Windows 4.0 if you happen to be using Borland’s command-line tools. By default, the version of tlink32.exe included with Borland C++ v5.0 marks 32-bit executables with an expected version number of 3.1. You have to use the “-V4.0” option to correct that. Also, although Borland’s resource compiler (brc32.exe) has an option to set the expected version number of a module, it appears to not actually work — you must use “-V4.0” with the linker to get the module marked correctly.
Summary
Processing WM_CAPTURECHANGED is the cleaner of the two methods and certainly the preferred choice if your application is targeted for Windows 4.0 or above. However, if you need to support pre-version 4.0 code or mixed-version code, WM_CANCELMODE is the better solution.
References
Otala, Tapani J. “Creating Dual-Version Apps for Win95/Win3.x,” Windows Developer’s Journal, October 1996. Uses Pietrek’s information to dynamically change a module’s expected Windows version number at runtime.
Pietrek, Matt. Windows 95 System Programming Secrets. San Mateo, CA: IDG Books, 1995. Pages 226-231 in Chapter 4 describe the internal HWND structure. Offset 0x2C (Pietrek calls this “dwFlags”) is where the 4.0 version stamp flag is stored. The value of the flag is 0x00400000.
Richter, Jeffrey. Advanced Windows. Redmond, WA: Microsoft Press, 1995. Chapter 10 explains how mouse capture works under Win32, including differences between the Windows 3.1 and Win32 implementations.
Chris Branch is a software engineer at FSCreations, Inc. in Cincinnati, Ohio. He can be reached at [email protected].