¶File dialogs break in capture mode with UScreenCapture driver
If you are using VirtualDub with the UScreenCapture driver to capture video from the desktop screen, you may run into problems with the Set Capture File option in VirtualDub not bringing up a file dialog, at least on Windows 7. This appears to be due a problem within v2.0.14 of UScreenCapture.ax which also reproduces with Microsoft GraphEdit. I am trying to contact the vendor for a resolution, but have not gotten a reply as of yet.
VirtualDub's file dialogs fail to come up because of an internal error within the Windows common file dialog. The specific failure path is GetSaveFileName() -> DialogBoxIndirectParam() -> CreateWindowEx(ComboBoxEx) -> SetWindowSubclass() -> GlobalAddAtom(). That in turn is caused by UScreenCapture's worker thread calling SetProcessWindowStation() with a handle to WinSta0 that does not have the WINSTA_ACCESSGLOBALATOMS access right. This only happens while the capture driver is running, but as it pretty fundamentally borks UI process state, it's not something I can easily work around safely.