CComActivator::DOCreateInstance throws access violation in 32-bit code.

Question

question

NormanViljoen-5702 asked May 12, '22 RLWA32-6355 commented May 19, '22

CComActivator::DOCreateInstance throws access violation in 32-bit code.

Hi All,

I am seeing an exception running 32-bit code using CComActivator::DOCreateInstance. This call causes an access violation when invoking CoCreateInstance to instantiate a device enumerator object in a 32-bit DLL. The same code compiled 64-bit invoking the same enumerator in a 64-bit DLL works perfectly.

IMMDeviceEnumerator* pDeviceEnumerator = NULL;
CoCreateInstance(__uuidof(MMDeviceEnunerator), NULL, CLCTX_INPROC_SERVER, __uuidof(IMMDeviceEnumerator), (LPVOID*)&pDeviceEnumerator);

push 0
push eax
push 1
push 0
push dword ptr [ebp+10h]
mov ecx, ebx
call combase!CComActivator::DOCreateInstance
(3bc.1cf4): Access violation - code c0000005 (first chance)

BTW. The calling convention used in the DLL for all the enumerator methods is STDMETHODCALLTYPE, which is the required calling convention through class inheritance. What am I missing here? Is it a calling convention problem? Any ideas?

Sincerely,
Norman

windows-api

· 14

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

RLWA32-6355 · May 12 at 07:32 PM

(3bc.1cf4): Access violation - code c0000005 (first chance)

The debugger sees a first chance exception before the application being debugged. A first chance exception isn't necessarily fatal if the exception is handled by the application. Was there a subsequent crash due to an access violation or some other problem?

I could not reproduce the problem using a console application. I assume that when you talk about 32-bit and 64-bit DLLs you are referring to the in-process COM server.

0 ·

NormanViljoen-5702 RLWA32-6355 · May 12 at 08:40 PM

Hi,
Thanks for the response. The DLL is a custom audio enumerator available as 32-bit and 64-bit. A client application has no problem accessing the 64-bit DLL, but a 32-bit client throws an exception when trying to access the audio enumerator methods. For instance the 32-bit wmplayer accessing the 32-bit endpoint audio enumerator DLL fails while the 64-bit wmplayer works correctly with the 64-bit audio enumerator DLL. Note that the custom audio enumerator C++ source code is identical for both 32 and 64 bit. Also in case you are wondering the DLL is in the correct folders. (\Windows\SysWOW64 for the 32-bit dll and \Windows\System32 for the 64-bit dll)

Anyhow, I created a simple C++ application to help debug the problem - all that it does is use CoCreateInstance to return an instance of the enumerator then call the various methods like get the EnumAudioEndpoints and GetDefaultAudioEndpoint and access the audio device and endpoint methods. In 64-bit everything everything works fine. But the 32-bit code fails in the CoCreateInstance call.

Furthermore, I also tested the endpoint enumerator DLL by bypassing COM and dynamically loading the DLL to call the GetTSAudioEndpointEnumeratorForSession directly to get the enumerator instance and call all the methods - this works without issue for both 32-bit and 64-bit. Ultimately, It is when accessing the 32-bit audio endpoint DLL via COM that there is a problem.
Thanks,
Norman

0 ·

RLWA32-6355 NormanViljoen-5702 · May 12 at 10:19 PM

Maybe the problem is in the class factory code since that would be the first place that CoCreateInstance would go. Of course, that's just a guess.

0 ·

JunjieZhu-MSFT · May 13 at 07:31 AM

In CoCreateInstance , Have you tried use CLSCTX_LOCAL_SERVER ?

0 ·

NormanViljoen-5702 JunjieZhu-MSFT · May 13 at 12:55 PM

I am using CLSCTX_INPROC_SERVER because the audio enumerator must run in the same context as the caller. Doing this lets my test application emulate what the audio clients are doing. Nonetheless, for interest's sake changing to CLSCTX_LOCAL_SERVER causes both 32-bit and 64-bit to fail with error code: 0x80040154.

0 ·

RLWA32-6355 · May 13 at 01:01 PM

Can you share your code with us so that we have some visibility into what is happening? You need not provide your implementation of interfaces since the problem seems to be the COM object instantiation. For test purposes interface implementations could simply return E_NOTIMPL.

0 ·

NormanViljoen-5702 RLWA32-6355 · May 16 at 01:05 PM

Hi,
Below is a concise version of the code. In 32-bit executables the exception occurs in the "App code" when CoCreateInstance returns. In the debugger I can see that GetTSAudioEndpointEnumeratorForSession is called. This problem does not occur in 64-bit.
Thanks,
Norman

// DLL code
/#define DLL_EXPORT(type) extern "C" __declspec(dllexport) type
DLL_EXPORT(HRESULT) GetTSAudioEndpointEnumeratorForSession(In DWORD SessionId, Out IMMDeviceEnumerator **ppEndpointEnumerator)
{
HRESULT hr = E_NOINTERFACE;
*ppEndpointEnumerator = NULL;
return hr;
}

// App code
int Test() {
HRESULT hr;
hr = CoInitialize(nullptr);
if (hr == S_OK) {
IMMDeviceEnumerator deviceEnumerator = NULL;
hr = CoCreateInstance(__uuidof(MMDeviceEnumerator), NULL, CLSCTX_INPROC_SERVER, __uuidof(IMMDeviceEnumerator), (LPVOID )&deviceEnumerator);
...
}
...
}

0 ·

RLWA32-6355 NormanViljoen-5702 · May 16 at 01:11 PM

What I had hoped was for you to share the relevant COM server code so that we might attempt to recreate the problem. Without some code from your custom enumerator we are left to guess at possible causes.

0 ·

Show more comments

RLWA32-6355 · May 16 at 02:31 PM

I haven't created a custom audio enumerator myself so I guess I don't understand enough about the topic to assist you further.

0 ·

NormanViljoen-5702 RLWA32-6355 · May 16 at 02:45 PM

Nonetheless, thank you for spending time looking at this and responding.

0 ·

RLWA32-6355 NormanViljoen-5702 · May 16 at 02:55 PM

One more thought -- Have you tested on multiple versions of Windows (8.1, 10, 11) and different builds of Windows 10/11? Can the problem be isolated to a particular environment?

0 ·

Answer 1 · 2022-05-17T10:56:03.91Z

NormanViljoen-5702 answered May 17, '22 RLWA32-6355 commented May 19, '22

I found the solution to this. The 32-bit call is expected to use the __stdcall calling convention. But using this convention causes the CoCreateInstance to not find the GetTSAudioEndpointEnumeratorForSession. To fix that I used a pragma:
#pragma comment(linker, "/export:GetTSAudioEndpointEnumeratorForSession=_GetTSAudioEndpointEnumeratorForSession@8")
Now the 32-bit code works as well as the 64-bit code.

· 3

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

RLWA32-6355 · May 17 at 11:03 AM

This strikes me as a documentation bug. The documented prototype for the exported function does not specify the use of the __stdcall calling convention. It would make no difference for 64-bit builds since calling convention specifications are ignored for them.

Then again, if the code that calls GetProcAddress for the exported function is incorrect the problem is bigger than mere documentation.

0 ·

NormanViljoen-5702 RLWA32-6355 · May 19 at 04:14 PM

As I have mentioned before, bypassing COM completely and loading the 32-bit audio endpoint DLL using loadLibrary and then getting the GetTSAudioEndpointEnumeratorForSession with GetProcAddress I could access the endpoint methods without a problem. Anyhow, using the #pragma solved the COM problem for 32-bit and now 32-bit clients like wmplayer work correctly. So it must be a Documentation issue where the 32-bit detail is missing. Perhaps, since 32-bit may be considered legacy it is just ignored :)

0 ·

RLWA32-6355 NormanViljoen-5702 · May 19 at 04:17 PM

Interestingly, if you use a def file to export GetTSAudioEndpointEnumeratorForSession there are no problems in either 64-bit or 32-bit. Maybe that should be in the documentation.

0 ·

question