Current version

v1.10.4 (stable)


Main page
Archived news
Plugin SDK
Knowledge base
Contact info
Other projects


Blog Archive

Working around display brain damage in Windows Vista

I've been struggling with video display issues in VirtualDub under Windows Vista for a while now, as some of you may know. I hit a couple of snags during the beta, one of which was due to a DirectDraw implementation issue in the OS that was fixed in RTM. 1.6.17 works decently well in Vista, fortunately. However, as I've optimized and reworked the display code in 1.7.x, I'm finding that I'm hitting a lot of weird issues in Window Vista again that I wasn't seeing in Windows XP. I spent part of last weekend fighting these again in another fit of frustration over things not displaying when they should.

When I see the exact same issues on two machines running Vista, one with an NVIDIA card and one with an ATI card, I'm inclined to believe it's Microsoft's fault.

The first problem, which I've mentioned before, has to do with DirectDraw hardware video overlays -- these are essentially secondary displays that are composited on top of the main one in the video scanout hardware itself. Yeah, yeah, Microsoft's been saying that video overlays are outdated... but they're the only widely available way to do hardware YCbCr color conversion for commonly used formats without requiring 3D pixel shaders of some sort. You'd be hard pressed to find a system out there with a resolution greater than 800x600 that doesn't support a YUY2 overlay. Well, the problem is that Vista will happily let you create a hardware overlay surface, populate it, and show it -- without actually displaying anything. Your program thinks its happily displaying video, when the user is actually seeing green, magenta, or whatever you use for your colorkey. Lame. I worked around this in VirtualDub 1.7 by calling DwmIsCompositionEnabled() if it is available, and forcibly disabling overlays if the DWM is compositing.

The second problem is more insidious. For various reasons, I'm moving the display code to a separate thread in 1.7.2, and this is exposing a lot of weird threading issues in Windows, like the HTTRANSPARENT issue I mentioned earlier. Well, another problem I hit is that the DWM doesn't seem to consistently update its composition tree when you have a child 3D window in another thread -- you can call Present() in Direct3D or SwapBuffers() in OpenGL, and nothing shows up. In fact, you get junk from underneath the window. I beat my head against the desk for hours trying to figure this out, and made the following conclusions:

The solution I finally came up with was to call SetWindowPos() with the SWP_FRAMECHANGED message after the first call to Present() or SwapBuffers(). This seems like an utterly bogus solution, and I see a frame of garbage whenever the D3D or OpenGL minidriver reinitializes, but in the absence of any better solution or any diagnostics to determine what's really going wrong, it's the best I can do. Sigh.

I think the most astonishing part to me is how Microsoft can form a movement to get applications "Vista compatible" -- when in reality what they've done is broken a lot of apps and asked the vendors to pick up the pieces. Sure, some apps were doing really broken things, but I'm just trying to use Direct3D to display video....


This blog was originally open for comments when this entry was first posted, but was later closed and then removed due to spam and after a migration away from the original blog software. Unfortunately, it would have been a lot of work to reformat the comments to republish them. The author thanks everyone who posted comments and added to the discussion.