¶One video codec to break everything
How does one video codec break every other video codec in the system?
Simple, really.
The way that video codecs work in Windows is that each one is queried in turn and asked if it supports the compressed video format. If a codec says "yes," then that codec is used to decode the video. Otherwise, Windows proceeds to the next video codec down the line, until it either finds one that can handle the format or runs out of video codecs. The details are different between Video for Windows and DirectShow, but the general idea is the same. This provides a great deal of flexibility, because a video codec can decode more than one format. Some MPEG-4 video codecs, for instance, can helpfully also decode uncompressed YCbCr formats that are difficult to find dedicated codecs for, like YV12.
It also adds a bit of fragility, in that occasionally a video codec's decompression query routine acts as follows:
Q: Do you support video format....
A: Yes!
Q: Uh, I didn't tell you the video format. You really support this?
A: Yes!
The errant video codec then proceeds to capture every video format that comes its way, and happily barfs whenever it gets any format other than the one it was supposed to handle. Since the shell media extension in Windows XP uses video codecs to try to get basic information and thumbnails for videos, Windows Explorer can now crash if you hover over the wrong videos. And guess what... the common File-Open dialog is based on an Explorer view, so it too can crash in any program.
As a result, one bad video codec can cause every other video codec in the system to fail to function. So far, I have encountered three video codecs that do this. It's even more fun when you realize that querying a codec involves loading it, so a codec that crashes on load due to Data Execution Prevention (DEP) crashes, anti-debugger checks, etc. can cause even more... fun.
For those of you who are considering or have worked on plugin APIs, this has some implications for API design. Querying every codec provides maximum flexibility to the video codecs, but also increases the fragility of the system -- if anyone goofs up, the whole thing breaks. Now, none of the three video codecs I've found so far were actually trying to decode multiple formats, but did so unintentionally. By having plugins provide some basic information that the API host can use to pre-filter queries, the system becomes more reliable, as it is much easier to make a single routine in the host reliable than a series of checks in all third-party plugins. It's also faster since you're not loading a ton of DLLs every time.
Those of you who read Raymond Chen's The Old New Thing blog may recognize the above problem as a variant on his "sure, we do that" story. Happily, the workaround is similar. What VirtualDub does is that it eschews use of the VFW ICLocate() function and instead manually enumerates and opens the video codecs, so it can directly control the query process. It then calls ICDecompressQuery() with a bogus compressed format and sees if the video codec bites. If the video codec takes the bogus format, then VirtualDub only uses that video codec if the compressed format is one that the codec says it can compress to; otherwise, it bypasses the codec and keeps going. You'll know if this happens because VirtualDub will display a warning on opening a video file with the name of the video codec that has the bug.