¶Working around display brain damage in Windows Vista
I've been struggling with video display issues in VirtualDub under Windows Vista for a while now, as some of you may know. I hit a couple of snags during the beta, one of which was due to a DirectDraw implementation issue in the OS that was fixed in RTM. 1.6.17 works decently well in Vista, fortunately. However, as I've optimized and reworked the display code in 1.7.x, I'm finding that I'm hitting a lot of weird issues in Window Vista again that I wasn't seeing in Windows XP. I spent part of last weekend fighting these again in another fit of frustration over things not displaying when they should.
When I see the exact same issues on two machines running Vista, one with an NVIDIA card and one with an ATI card, I'm inclined to believe it's Microsoft's fault.
The first problem, which I've mentioned before, has to do with DirectDraw hardware video overlays -- these are essentially secondary displays that are composited on top of the main one in the video scanout hardware itself. Yeah, yeah, Microsoft's been saying that video overlays are outdated... but they're the only widely available way to do hardware YCbCr color conversion for commonly used formats without requiring 3D pixel shaders of some sort. You'd be hard pressed to find a system out there with a resolution greater than 800x600 that doesn't support a YUY2 overlay. Well, the problem is that Vista will happily let you create a hardware overlay surface, populate it, and show it -- without actually displaying anything. Your program thinks its happily displaying video, when the user is actually seeing green, magenta, or whatever you use for your colorkey. Lame. I worked around this in VirtualDub 1.7 by calling DwmIsCompositionEnabled() if it is available, and forcibly disabling overlays if the DWM is compositing.
The second problem is more insidious. For various reasons, I'm moving the display code to a separate thread in 1.7.2, and this is exposing a lot of weird threading issues in Windows, like the HTTRANSPARENT issue I mentioned earlier. Well, another problem I hit is that the DWM doesn't seem to consistently update its composition tree when you have a child 3D window in another thread -- you can call Present() in Direct3D or SwapBuffers() in OpenGL, and nothing shows up. In fact, you get junk from underneath the window. I beat my head against the desk for hours trying to figure this out, and made the following conclusions:
- That the DWM eliminates flicker is BS. It can sometimes make it worse, because changes in the window hierarchy that were ordinarily atomic can now be visible. Formerly, if you deleted a child window and created a new one in its place, you were OK because the message pump didn't run in between and WM_PAINT couldn't be sent to any windows on the thread to cause flicker. Now the DWM can rebuild the composition tree in between and add flicker to a previously solid display.
- Calling GetDC() causes a Direct3D-based window to stop updating. Okay, sure, I can understand not wanting to support mixed GDI and D3D rendering at the same time on the same window, but it'd be nice to have some better sort of fallback mechanism than just ignoring all of my Present() calls -- at least show something, even if it's slow and flickery. Oh, and it's really hard to figure out if a window is visible without an HDC (I use RectVisible() for this). I ended up calling GetDCEx(hwnd, NULL, 0) on the parent window instead.
- You can call BeginPaint() safely, although I suspect you have to be really careful about what you do with the HDC.
- Doing a FillRect() every frame makes the D3D rendering show up... not that I'm about to ship that solution.
- WS_EX_COMPOSITED and WS_EX_LAYERED do squat.
The solution I finally came up with was to call SetWindowPos() with the SWP_FRAMECHANGED message after the first call to Present() or SwapBuffers(). This seems like an utterly bogus solution, and I see a frame of garbage whenever the D3D or OpenGL minidriver reinitializes, but in the absence of any better solution or any diagnostics to determine what's really going wrong, it's the best I can do. Sigh.
I think the most astonishing part to me is how Microsoft can form a movement to get applications "Vista compatible" -- when in reality what they've done is broken a lot of apps and asked the vendors to pick up the pieces. Sure, some apps were doing really broken things, but I'm just trying to use Direct3D to display video....