¶Exception handling on AMD64
I ran into an annoying problem yesterday while rewriting some of VirtualDub's assembly language routines for AMD64.
Take this innocuous looking routine, for instance:
.code
trap proc public
sub rsp, 256
int 3
add rsp, 256
trap endp
end
It just fires the application crash handler with a breakpoint exception. Well, actually, not quite. What it actually does is terminate the application. The reason has to do with the way exception handling works on AMD64 Windows.
In Win32, active exception handlers are kept together in a singly-linked list, pointed to by the Thread Environment Block (TEB). The TEB, is in turn, set as the base of the FS selector, so that FS:[0] is the head of the exception handler list. When an exception occurs, such as from a C++ throw statement, the OS walks the list, looking for a handler that understands the exception. When it finds one, it then executes the handler, and optionally executes the handlers on the list down to that point again to unwind the stack. If no handler accepts the exception, the OS invokes the unhandled exception filter set by the application, or UnhandledExceptionFilter() if none is set. VirtualDub plugs in its own handler, which unwinds the app's context stack to show the "friendly" dialog, then dumps the disassembly and stack to create the crashinfo.txt file.
Matt Pietrek's A Crash Course on the Depths of Win32 Structured Exception Handling article is the best reference I know of on Win32 SEH; Chris Brumme's blog entry The Exception Model fills in a lot of subtleties missed by Pietrek, particularly what happens when the EH process fails. Sadly, the Platform SDK documentation is woefully inadequate in this area.
I also abuse the FS:[0] SEH chain to hold the stack pointer when I need to reuse ESP as an eighth register, but that's a different story.
AMD64 dumps this method of EH and switches to table-based exception handling. Instead of manually setting up an exception handling chain in code, instead the compiler emits tables indicating how program locations correspond to functions, which exception handlers to use, and how to unwind. The benefit is that no explicit linking/unlinking is required and thus the runtime overhead of try/catch blocks is minimal (there is still some overhead in making sure there are clear exception handling points in the code). The disadvantages are that the exception handling tables can be rather large, and that they must exist for every function that may need to be unwound when an exception is fired.
Let me repeat that again: the tables must exist for every function that may need to be unwound. Not just functions that throw, or functions that have explicit or implicit try/catch blocks. Every function that has a stack frame needs an EH structure; the exception is functions that do not allocate stack space and thus have no stack frame (leaf functions). The assembly function above has a stack frame but has no exception handling tables declared. So the OS attempts to unwind past the function, fails, and then simply terminates the application. No exception handler fires, no crash dump is produced, and the program simply vanishes.
This, frankly, is unacceptable. Well, one way to fix it is to insert directives so the necessary tables are created:
.code
trap proc public frame
sub rsp, 256
.allocstack 256
.endprolog
int 3
add rsp, 256
trap endp
end
This allows the OS to unwind the function, and exception handling works. Thankfully, you only need to annotate instructions that alter the stack layout, not every instruction. What I don't like is how easy it appears to accidentally sabotage the exception handling system so that any subsequent exception is uncatchable. All it takes for one driver to load and run asm code without the correct tables -- if it crashes anywhere in that code, the application will simply die, with no hope for a crash report to figure out what happened. Given the number of problems I've seen in the past with third-party codecs and libraries, this scares me.
What I need is a hook that gets executed when the stack unwind fails, but so far I haven't found one yet. There is a facility to plug in a function to produce tables for dynamically generated code, which is a possibility; in theory I could write a hook that creates fake RUNTIME_FUNCTION structures for the code that crashes and point the OS to my crash handler. The huge problem with this is distinguishing a valid crash in a leaf function from a crash that won't unwind. A third possibility is to figure out where the OS code is that terminates the app on an unwind failure and patch it, but I'm sure the Windows guys just love dealing with that kind of code.