How to troubleshoot DCOM 10009 error logged in system event?

Symptom:

 

Sometimes, we may see below DCOM 10009 errors in our system event, or we may receive the exception code 0x800706ba in our DCOM client application:

Event Type: Error
Event Source: DCOM|
Event Category: None
Event ID: 10009
Date: 2010-2-22
Time: 10:02:07
User: N/A
Computer: <Computer Name>
Description:
DCOM was unable to communicate with the computer <Target Computer Name> using any of the configured protocols.

Simply speaking, DCOM 10009 indicates that the DCOM client located on this <Computer Name> can’t communicate with the DCOM|COM+ server located on that <Target Computer Name>.

How this happens:

 

When a client calls CoGetClassObject or CoCreateInstance (CreateObject or new in VB) to activate a component, the COM runtime contacts its local SCM COM activator (RPCSS service) in order to launch the corresponding COM server that will host the desired component.

 

If the component is remote, the local RPCSS service will forward the request to the RPCSS service of the remote machine.

 

However, if the RPCSS service of remote target server is not available, DCOM 10009 will be logged locally to notify administrator. The typical call stack is as follows:
ChildEBP RetAddr

006df364 757f8b72 ADVAPI32!ReportEventW-> then DCOM 10009 is logged.

006df3a0 757f6542 rpcss!LogRemoteSideUnavailable+0x63 ->here we don’t found the server.

006df40c 757f6781 rpcss!CRemoteMachine::Activate+0x294

006df648 757f6861 rpcss!RemoteActivationCall+0xf2

006df664 757edb5c rpcss!ActivateRemote+0x8e

006df6c0 757d629c rpcss!Activation+0x343

006df718 757d7680 rpcss!ActivateFromProperties+0x1c2

006df724 757d76c0 rpcss!CScmActivator::CreateInstance+0xd

Why this happens:

 

Totally speaking, the reason why DCOM 10009 is logged is that: local RPCSS service can’t reach the remote RPCSS service of remote target server. There are many possibilities which can cause this issue.

  • Scenario 1:
     The remote target server happens to be offline for a short time, for example, just for maintenance.
  • Scenario 2:
    Both servers are online. However, there RPC communication issue exists between these two servers, for example: server name resolvation failure, port resources for RPC communication exhaustion, firewall configuration.
  • Scenario 3:

Even though the TCP connection to remote server has no any problem, but if the communication of RPC authentication gets problem, we may get the error status code like 0x80070721 which means “A security package specific error occurred” during the communication of RPC authentication, DCOM 10009 will also be logged on the client side.

  • Scenario 4:

The target DCOM |COM+ service failed to be activated due to permission issue. Under this kind of situation, DCOM 10027 will be logged on the server side at the same time.

 

How to troubleshoot:

 

As I mentioned above, there are many possibilities which can cause DCOM 10009 being logged. Some scenarios are normal while others are abnormal. We should deal with the DCOM 10009 in different ways.

  • · For scenario 1
    if DCOM 10009 is logged during the period of maintenance of remote target server, it’s an expected behaviour because the remote target server is offline. We can verify it by ping <remote server>
  • · For Scenario 2
    If the remote target server is online while DCOM 10009 is still logged, then we can perform below tests:

1) Ping <remote server> & IP Address of remote server to make sure network traffic is fine.

2) Telnet <remote server> 135 to make sure the Port 135 is not blocked by firewall.

3) If we have configured dynamic port for RPC communication, we can check what status the port resources are on both servers using below command:

netstat –anb

if all dynamic ports are used , or only few left, you can consider expanding the dynamic port.

How to configure RPC dynamic port allocation to work with firewalls

http://support.microsoft.com/?id=154596

DCOM port range configuration problems

http://support.microsoft.com/kb/217351/en-us

4) If all above tests are fine, we can use DTCPing tool to verify if the DCOM communication between two servers is fine.

How to troubleshoot connectivity issues in MS DTC by using the DTCPing tool
http://support.microsoft.com/kb/918331/en-us

Note: DTCPing tool is not specific for troubleshooting MSDTC issue, it can be used to troubleshooting all DCOM communication issues.

    • For scenario 3:
      Need capture more resources like Network traffic, or even iDNA trace to further investigate how this failure happens. 
    • For scenario 4:

Grant the specific account mentioned in DCOM event 10027 with corresponding permission through “Component Services MMC”

 

References:

FIX: You cannot obtain detailed error information about DCOM 10009 errors in Windows Server 2003
http://support.microsoft.com/kb/910695

Winston from APGC DSI Team