§ ¶The video level mess in Windows
I've gotten so many reports over the years about mismatched YCbCr video levels in VirtualDub and had to explain the situation so many times that I'm tired of it... even though it is important. The level situation in a simple video workflow often involves hardware and software from at least half a dozen vendors and therefore no one can unilaterally solve the problem, thus resulting in frustration for everyone. This is the issue that sums up the situation best, though:
- Launch Windows Media Player.
- Play a video.
- Go into Tools > Options > Performance, and change the video acceleration level from Full to None.
- Play the video again.
On some but not all systems, you will get different black and white levels between these two modes, meaning that software and hardware color conversion are inconsistent. Even worse, this only happens with videos where YCbCr decoding is active -- use a RGB uncompressed stream or a decoder that outputs RGB, and you won't have this problem. Regardless of who's correct, this leads to the unhappy situation where it is impossible to encode a video that displays properly on everyone's system, and pretty much anything you try will be wrong in some way. It also means that if you haven't noticed this particular issue, and were using your player to calibrate your pipeline, you may have done so incorrectly. Joy.
Compounding this issue is the "two wrongs make a right" problem -- you may have two stages in your workflow that are both using the wrong levels, but the two cancel out. An example is that if you have 16-235 input coming in, but your encoder is interpreting that as 0-255 and the decoder is also producing 0-255 while the display code is expecting 16-235. This works out great until you already have a video playing when you try to examine another one, at which time the second player drops out of acceleration mode because the hardware overlay is already in use, and the decoder attempts to convert YCbCr to RGB in software and whacks the black and white levels.
Ultimately, I think that only Microsoft is in a situation to fix this, since only it controls the APIs and has enough clout to spec out and propose/push a solution to the codec and driver vendors. There are multiple problems involved: formats whose levels aren't defined, formats that don't have a way to specify levels, and formats that are specified but for which modules are accidentally or even willfully violating the spec. I don't have high hopes for a solution soon, though, after what I saw happen to the Direct3D overlay API, which was an API that had level options defined from day one and also ended up being fatally broken from day one.
If MS were to fix it, it would only be as a new API (to allow backwards compatibility)
otherwise if they 'broke' the existing APIs to fix them, they'd also be breaking an untold
number of games, video players / editors, etc. and only current software in that list would
get patched...too bad for you if you're using an old version of AweSomePlayer DX 2005 and it
doesn't work anymore because the author only supports the 2010 version.
Having said that, microsoft could introduce a new API and make it a windows logo requirement that
any app using component video use it (for the next version of windows).
That would ensure backwards compatibility and get all the big software houses on board with the new API
(if such an API existed)
whitetiger - 21 11 10 - 15:16
Well, there is such an API: OpenGL/Direct3D + "write your own shaders". Quite a reliable way to stop drivers etc. from messing too much, as long as you don't mind the effort and compatibility issues with old hardware that involves.
(Note: I am biased as I am responsible for the OpenGL output in MPlayer that does this :-) )
Reimar - 21 11 10 - 20:08
You're right, the levels issue is a big mess -- most graphics driver and video renderer writers ignore the fact that video virtually always comes with 16-235 levels and happily display it as it was 0-255.
I don't quite get the part about video codecs though. These components do (and should) not care about levels at all -- they just encode whatever input is thrown at them, regardless of whether it's limited to 16-235 or not.
Anyway, the even bigger mess in all this YCbCr stuff is the color space issue, i.e. what exact dialect of YCbCr we're talking. There's BT.470, BT.709, the xvYCC variants of it, YCgCo and a dozen not-so-common others. BT.470 was the undisputed standard for years, so this is what every implementation supports -- but what about BT.709, the color space in which most HD content is delivered? I've yet to come across a video player that gets this stuff right.
And don't get me started about transfer characteristics, color primaries and white points -- I bet that no single consumer application (hardware included) cares about these.
KeyJ (link) - 22 11 10 - 04:59
> Well, there is such an API: OpenGL/Direct3D + "write your own shaders". Quite a reliable way to stop drivers etc. from messing too much, as long as you don't mind the effort and compatibility issues with old hardware that involves.
Yup, that's actually what I do in my Direct3D path. Of course, then you get complaints from people who think the other way is the right way... and unfortunately, for their setup, that's probably correct.
> I don't quite get the part about video codecs though. These components do (and should) not care about levels at all -- they just encode whatever input is thrown at them, regardless of whether it's limited to 16-235 or not.
They generally don't... until the conversion to RGB happens. You're assuming that the color conversion is happening in the presenter, when it can happen in the decoder instead. DirectShow will request an RGB format if you lose display acceleration for any reason, such as remote desktop on XP or moving the window to a downlevel secondary device. You'll also run into this problem if the decoder and presenter are the same device, such as an accelerated decoder or dedicated video output device. MJPEG is a big mess in this fashion, because of the mixes of cameras, software and hardware codecs, etc. You can get away with pass-through for your local workflow but if you're trying to interchange files you'll get burnt.
I think you can also get slightly singed if you try to pass full-range video through a codec that is constrained to clamp off values 0 and 255 as reserved. I've seen references to this in QuickTime, but not in VFW/DirectShow.
> Anyway, the even bigger mess in all this YCbCr stuff is the color space issue, i.e. what exact dialect of YCbCr we're talking.
Yup, although it's a lot less noticeable than your levels getting whacked since at least for BT.601 BT.709 luma is the same. Fortunately, there's been more movement on this front, such as with the establishment of the HDYC FOURCC as a BT.709 version of UYVY.
Phaeron - 22 11 10 - 16:08
DirectShow is deprecated, same with all legacy video renderers, Overlay Mixer, VMR7/VMR9, use Media Foundation and EVR renderer. MS might even remove all DirectShow support in Windows 8.
"It is the intended replacement for Microsoft DirectShow, Windows Media SDK, DirectX Media Objects (DMOs) and all other legacy multimedia APIs such as Audio Compression Manager (ACM) and Video for Windows (VfW)."
MF - 27 11 10 - 13:18
Uh, OK, but in the meantime we still have a significant user base using OS versions which cannot use Media Foundation, we still have low-level APIs that are underspecified or not working as intended, and codecs are still primarily being written to VFW or DirectShow.
Phaeron - 28 11 10 - 20:07
I'm sure that VfW or other legacy video apis will never be removed from Windows.
Tell one "important" feature MS had ever removed from Windows. They are so focused on backward compatibility that this will never happen.
Kelden - 30 11 10 - 03:33
Hardware concerns have forced Microsoft to remove some subsystems, the most notable being support for 16-bit DOS apps in 64-bit Windows. DirectDraw is also mostly removed at this point anyway, working only in emulation mode with WDDM drivers.
Phaeron - 30 11 10 - 15:05
As you see DD was not removed, it's emulated now. ;)
But you are right. They removed 16 bit Support on x64 systems.
But that's not one api, that's a whole subsystem.
I think the reason is that it wasn't very important anymore. But when you look at VfW or DirectShow, there are a lot of applications which uses them.
Kelden - 30 11 10 - 20:09
Interesting how MS is basically deprecating all of DirectX except for D3D. Funny how VfW still works 100% in Windows 7 x64, MS said it would be gone after Windows 2000 :P. Lets hope "Media Foundation" is a somewhat clean and straightforward API. Oddly I don't think I have ever run into the video level problem, but I'll keep my eye out for it.
Chris - 02 12 10 - 17:27
"The options to use the overlay mixer, video mixing renderer (VMR-7) or high quality mode (VMR-9) has been removed."
So you say Microsoft doesn't remove features? Think again.
MF - 05 12 10 - 22:33
I just played around with some sample Media Foundation capturing programs in the WinSDK. Bad news, only supports USB Video Class 1.1 capture devices (aka web cams). If this plans on replacing DirectShow any time soon, its going to need a better capture device support. Otherwise we are back to using wrappers (which don't exist yet) to support most video capture cards.
Chris - 06 12 10 - 11:24
Please consider adding Direct2D support for UI rendering since MS had the brilliant idea of deprecating GDI/GDI+ & making everything GDI/GDI+ run in software in Vista/7/whatever future Windows, horrendously slow and sluggish GUI for all applications using GDI/GDI+.
"Direct2D offers high quality and fast performance while maintaining interoperability with GDI/GDI+ APIs and Direct3D/DirectDraw APIs. It can take advantage of hardware acceleration through compatible graphics cards."
You mentioned it before here, http://www.virtualdub.org/blog/pivot/ent..
"The main reason that Vista performs worse is that, despite all the hype about the new desktop hardware acceleration, the WDDM 1.0 display driver model introduced in Vista actually forces all GDI and DirectDraw rendering to occur in software on the CPU. Since most UI in Windows is based off one of those two APIs -- including Win32 UI and .NET's GDI+ based UI -- this results in significantly slower UI performance. Well, unfortunately, it doesn't seem that Windows 7 runs Aero Glass any better, and in fact on this system VirtualDub has trouble breaking around 25 fps due to the low desktop composition frame rate."
D2D - 08 12 10 - 07:50
A lot of the Windows Vista/7 performance issues apparently were video driver related. Tom's Hardware did an in-depth story on the issue complete with GDI benchmarks.
Chris - 08 12 10 - 12:41
Windows 7 does accelerate GDI if a WDDM 1.1 driver is available. It still does not accelerate DirectDraw, AFAIK. I believe Direct2D is just layered on top of Direct3D 10, so there's no real benefit to using it over regular D3D9/D3D10. Given that I already have a D3D9 driver that could probably be extended to D3D9Ex without much work, I don't see a point in doing D2D.
Phaeron - 08 12 10 - 15:11