x64 calling convention and the disappearing process syndrome
Raymond Chen describes the parameter passing aspect of the x64 calling convetions. But there is more than parameter passing to the calling convention. Exception handling is an important part of the calling convention.
A function that calls another function or needs to allocate stack space or requires exception handling (e.g. has a try statement) must have a prolog and an epilog. It also has to have an entry in a special function-table. The function table includes unwind information – information that enables the exception-handling routings to unwind the stack and undo the effect of the function prolog. In order for exception handling to work, there are limitations on function prolog and epilog.
The fun begins when a function does not have correct unwind information. If in addition to that, there is no debugger attached to the process, the system notifies the Win32 sub-system about the exception. The Win32 sub-system will simply kill the process. You will not see any Watson or JIT debugger dialog box. The process will just disappear.
This happened to me last week. I had an assembly thunk function that called some C++ code that had a race condition (which seems to happen only when a debugger is not attached). Debugging would have been much easier if I did not have an assembly thunk that did not play by the rules and did not have a function-table entry.