¶Enabling visual styles on an app breaks monochrome icon buttons
I finally got around to figuring out why VirtualDub's mark-in and mark-out control buttons don't display properly when comctl32 6.0 (visual styles) is enabled. It turns out to be a bug with icon handling on buttons in the new common controls library.
This is what a test app looks like, built against the regular common controls library (5.82):
The button on the left uses a 16 color icon, and the button on the right uses a monochrome icon. So far, so good.
Now let's try it with a comctl32 6.0 manifest to enable visual styles:
Oops. What happened to the right button?
It turns out that version 6.0 of comctl32 replaces the default button control class in the process when it loads, so that it can apply visual styles. That's fine. What's not so fine is that it has a bug with the way it handles the BM_SETIMAGE message for icons. I went spelunking in the disassembly and discovered that it attempts to measure the size of an icon as follows:
ICONINFO info;
if (GetIconInfo(hIcon, &iconInfo)) {
BITMAP bitmap;
if (GetObject(iconInfo.hbmColor, sizeof(BITMAP), &bitmap)) {
...
}
}
While this works fine for color icons, it's wrong for monochrome icons. The reason is a peculiarity in GetIconInfo(), which is that for monochrome icons it doesn't return separate mask and color bitmaps, but instead returns both the AND and XOR planes of the icon as a double-height bitmap in the hbmMask field. The result is that the v6 button code tries to query with a null bitmap handle, and when that fails it dumps the set image request on the floor. It turns out that those are the only two buttons on VirtualDub's position control that use monochrome icons; a long time ago, when I redid the icons to have a sort of 3D look, those were the two that looked bad even to me and so I left them as monochrome.
The fix? Just make the icons 16 color instead.
This is a good example of why you can't just turn on visual styles on an application: the new v6 common controls library is not 100% compatible and valid code can and will break. That doesn't mean you can't do it or that it will be hard to fix the problems, but it does have a non-zero cost. That goes double if you support plug-ins as I do, because then you have to worry about the plug-ins breaking as well.