¶The case of the PIX-only crash
PIX for Windows is a tool that comes with the Microsoft DirectX SDK for debugging Direct3D applications. I have a love/hate relationship with it, but when it comes to debugging rendering state problems, I haven't found anything that works better. NVPerfHUD is no good at tracking down single-frame glitches, and Intel Graphics Performance Analyzer (GPA), while surprisingly useful for debugging, doesn't expose full device state. Thus, I find myself using PIXWin a lot, even despite its shortcomings (its data presentation could use a lot of cleanup).
While trying to use it today, I ran into the annoyance of a PIX-only crash, meaning that the program would crash only when run with PIX instrumentation enabled.
Typically, I turn on the Direct3D debug runtime and make sure the program is debug runtime clean before trying to use PIX. The debug runtime fires debug asserts if it sees badness like invalid state, which is much easier to track down with the debug runtime than from mysterious crashes or error asserts in PIX. Unfortunately, that trick didn't work this time as the program ran cleanly outside of PIX and bombed inside of it, so I had to attach a debugger to the process while PIX had it hooked to try to diagnose the problem.
Long story short, the problem had to do with the extra code that the PIX hook runs when it instruments a process. PIX works by hooking the Direct3D entry points and running extra code to capture the D3D calls made by the program. Apparently, one of the things it does when D3D is initialized (Direct3DCreate9) is use Windows Management Instrumentation (WMI) to get some information about the graphics device. This in turn makes a blocking COM call to another thread, which then runs a message loop within the current thread to service messages while waiting for the other thread to run the call. This is needed to service other cross-thread calls back into the blocked thread and prevent deadlocks.
The other piece of the puzzle is a change that was made to COM starting with Windows Vista to allow WM_PAINT to be dispatched during blocking calls. IMO this is a bad idea as dispatching anything other than other sent messages (SendMessage) invites reentrancy problems, and that is true here as well. In my case, this caused a paint message to be dispatched on the display window that was still stuck in init in the Direct3DCreate9 call, resulting in an attempt to refresh the window using the 3D context that hadn't been inited yet. The workaround I applied was to add a reentrancy counter to block normal WM_PAINT handling on that window until init had completed.