¶Your video is now a garish purple blotch
Ever tried to do something with a video player, and gotten a fluorescent green or purple blotch where your video was supposed to be? Like take a screenshot, move it to a second monitor, or show it on TV out?
This is caused by the use of a hardware video overlay. This is a second video screen that is overlaid over the primary surface by the video hardware. Because the two are only generated at scanout, and even then sometimes only on the primary monitor, attempts to capture the video by screen capture will just give you the underlying window instead, even though you can see the video just fine.
The easiest way to work around this problem is to change the video acceleration options for your video player to use a lower setting, which usually disables use of the video overlay. A few applications, most notably DVD players, won't let you do this, in which case you'll either need to find a screencap application that hooks DirectX, or loop TV out into a video capture device, assuming your video card can display the overlay on TV out (some can't, or don't when the TV is the secondary monitor).
Activation of video overlays
Video overlays are created on Windows using the DirectDraw API, and only through that API. GDI and Direct3D need not apply. The overlay is exposed as a special type of surface; it can be locked and filled as any other video memory surface, but is rather special in that it can often be flipped independently of the main display. This means it can be easily updated in a tear-free manner by updating the back buffer and calling Flip(), without having to worry about the overhead of a blit when scheduling the flip. Because overlays are merged into the display without actually writing to the framebuffer, they can be shown, hidden, and moved extremely cheaply, much faster than even an accelerated blit.
The other main advantage of hardware video overlays is that they usually support YCbCr formats for image data, doing the conversion on the video card itself for free. On slower systems, this was a significant cost to video decoding and helped performance significantly. On modern CPUs it's not nearly so bad, and on a 3D card one or two pixel shaders (or a series of fixed-function blending hacks) will do the job equally as well. However, although you'll still find systems that can't do hardware 3D, you'd be hard pressed to find ones that can't support a hardware video overlay.
If you're a programmer and want to play with overlays, the DirectDraw Test application (ddtest.exe) from the DirectX SDK is the best way to do so. Just make sure you don't expand the overlay full-screen and make it impossible to see anything.
The colorkey
The reason for the purple blotch where the video should be in a desktop screenshot of a video player is that it's the colorkey for clipping the overlay. With a rare exception here and there, overlays usually can't be directly clipped to anything more complex than a rectangle, or perhaps not at all. This is a problem when windows or title graphics need to overlap the overlay, so most hardware supports a destination colorkey, where the overlay is only shown if a specific color is drawn on the underlying primary surface. Any windows on top don't draw colorkey-colored pixels — hopefully, at least — so the overlay is hidden there and gets covered like the rest of the player window. Ugly and seldom-used colors are chosen for the colorkey to minimize collisions. I hate seeing flashes of nasty magenta or cyan when manipulating a video window, so VirtualDub uses #008000 dark green instead — which occasionally causes some interesting effects when I open Task Manager on top or stop the program at a breakpoint and look at some code comments where the overlay is still active.
Using a colorkey for clipping does have the nice benefit of clipping 100% correctly, whereas blits frequently don't since the complex region gets cut up into subrects and the subrects then have coordinates rounded to integers, leading to seams. Direct3D uses a special blit command for its Present() that preserves enough data to avoid the rounding errors, but for some reason this capability wasn't added to DirectDraw's Blt().
If you know DirectDraw programming and have an unsuspecting buddy, you can do some wicked practical jokes with a full-screen overlay surface that uses a white destination colorkey to show images in the background of random windows. Not that I've ever done that before.
Quirks
With most hardware, you only get one overlay. If you try opening two players that both want to use the overlay, the second player will either fail or blit instead.
Overlay hardware generally only supports YCbCr color spaces. You might find a video card somewhere that supports an RGB overlay, but I don't know of any modern video cards that do. On the other hand, supporting UYVY and YUY2 overlays will work on just about everything. You do have to cover both, because I've seen ones that only support one and not the other.
Image interpolation is often not up to par on the overlay. Chroma is not fully interpolated, and I used to curse at NVIDIA constantly for disabling vertical interpolation entirely on later versions of the TNT2 drivers. ATI uses an edge-sensitive interpolation algorithm that can show ringing around the borders of the image.
Some video drivers play goofy tricks with overlays, like replacing a secondary TV-out monitor with the overlay when both are active. There isn't a way to detect this, although newer video drivers make the behavior configurable. The overlay also frequently has different color adjustment controls than the primary surface, although again these are regrettably not available programmatically.
On some video cards, there is a limit to how small the overlay can get horizontally, instead of how wide. There isn't a speed problem with making the overlay larger, since it's only being merged at scanout, and isn't actually writing over a huge number of pixels in the framebuffer. However, making it narrower is a problem because it means the video hardware has to fetch pixels faster over a shorter amount of time. This leads to the strange result that the overlay can be made full screen, but can't be made too small because the bandwidth requirements become too high. I have an old Toshiba Tecra laptop with a Chips & Technologies 65550 video chip that shows artifacts in the overlay is made too small.