Get in the loop: Control your VB apps' window messages with subclassing

To appreciate the benefits of subclassing, you first need to know how your VB applications handle window messages. Once you do, you can jump into the message queue and gain an impressive degree of control over how your apps behave.

Subclassing refers to the tried-and-true technique of tapping into your VB application's window message queue to directly handle messages. Properly implemented, subclassing can give you unparalleled control over your application. In this article, I'll introduce the technique and explain why you'd want to use it. Before we get started, though, let's cover some necessary background and make sure you've got the big picture.

Just like passing notes in class
The first important thing to understand is that Windows communicates with a GUI application by placing messages onto the application's message queue. These numeric messages include both:
  • Standard messages representing any system event from a mouse movement or key press to a pending disk write.
  • Custom messages that can be private to an application and represent anything the programmer wants.

A graphical Windows application must therefore implement a message loop to monitor the queue and handle messages as it receives them from the operating system.

The second thing to keep in mind is that not only is a form a window, but with a few exceptions, every GUI element from a label to a command button is a window as well. When a message for a particular window, like a mouse click on a command button, is received, the message loop notifies the window by calling its window procedure. The window procedure then takes some action based on the content of the message. In the case of our command button, that would be repainting the button in a "clicked-down" state and probably calling some other code.

The window procedure acts on any relevant incoming messages that is received in a timely fashion (like a mouse click over our hypothetical command button). If a message isn't relevant—for example, when the user presses an alphanumeric key while our command button has the focus—the window procedure can essentially ignore it.

Fascinating, but what's that got to do with me?
Obviously, you don't have to write a message loop or handle window messages in your VB apps. VB takes care of all this for you and translates some of the standard messages into equivalent events, such as raising a click event for that imaginary command button I keep talking about. It also sends other messages on your behalf in response to your manipulation of control properties like AutoRedraw or Enabled. That's probably elementary stuff for some of you. But at one time, as the old-timers will recall, event-driven programming like this was quite the revolutionary idea.

Now, VB doesn't map all possible standard messages into events, just the important ones like mouse clicks and key presses. It simply discards the others. That's usually a good thing; after all, when was the last time you really needed to know when a pending disk write was about to be committed? The thing is, some of those messages can be useful to a programmer in certain situations, so it helps to know how to listen for certain window messages.

The habits of successful windows procedures
If written in VB, the declaration for a window procedure would look like this:
Function VB_WindowProc(ByValhWnd As Long, ByValuMsg As Long, _
 ByValwParam As Long, ByVallParam As Long) As Long

From left to right, here's what those parameters mean:
  • hWnd—This is the window handle of the message's destination window. (Remember that in Windows-speak, virtually every GUI element is considered a window.)
  • uMsg—This is the actual message.
  • wParam and lParam—This is additional information that varies with the meaning of the message. For example, it could be the coordinates of a mouse click.

A classic illustration of the usefulness of subclassing is displaying a custom context menu when a user right-clicks on a text box. Ever tried to pull that off with code in your textbox's MouseUp event? Foolish mortal, the system's context menu takes precedence over yours and will always be displayed before your custom menu.

Check out Listing A for a solution. First, call Hook to set up VB_WinProc as the default windows procedure for your text box (pass the textbox's hWnd property) using the SetWindowLong API function. VB_WinProc then listens for and intercepts WM_RBUTTONUP messages, which indicate that a user has released the right mouse button over the text box. When a right click is detected, the custom menu is shown via a call to some public method on your form.

You should notice two important things about the example in Listing A. First, the UnHook routine must be used to restore the original window procedure to the subclassed control before the form containing it is destroyed. Second, it's not enough for VB_WinProc to look like a window procedure. It has to act like one as well, immediately passing along any messages it isn't interested in to the default window procedure and handing the return value back to the system. Failure to follow either of these rules usually results in a swift death for your application via any number of nasty memory protection errors.

Breaking the message code
The process of trapping other messages bound for other controls is similar to the one used for detecting right mouse clicks. The only trick is figuring out what uMsg values equate to which messages. Figure A lists the names and decimal equivalents for a few of the more important window messages. For the meaning of other messages, I'd recommend that you check out MSDN or any of Dan Appleman's Windows API reference books. Once you've determined the message you want, you can find the appropriate VB constant using the API Viewer applet that ships with Visual Studio.

Figure A
Selected window messages and their VB declarations

There's still another side to this story: altering a window's behavior by sending it messages. That'll be the subject of my next article.

Editor's Picks