MSDTC communication is not working on a Windows 2008 Cluster when Incoming Caller Authentication or Mutual Authentication is required
In the design change between Windows 2003 and Windows 2008, MSDTC running on a cluster should now be able to take advantage of all available MSDTC authentication option when servers are running on the same domain as their MSDTC client servers.
There is however a problem which causes this supported scenario to fail.
Let us take for scenario where MSDTC is running in a SQL (Service or application) group and this SQL group was originally configured as part of the SQL Server setup process. The resource name of Network Name resource is probably called something like "SQL Network Name (MyClusterSQL1)" (where MyClusterSQL1 is the actual DNS name and the name used in the network name properties). Due to a problem in MSDTC this mismatch between the resource name and the network name itself causes the authentication (whether incoming caller authentication or mutual authentication) to fail.
If you are facing this scenario we can recommend the following change which will rename the "SQL Network Name (MyClusterSQL1)" network name resource to “MyClusterSQL1”. Below is an example of how to make this change:
Cluster.exe res "SQL Network Name (MyClusterSQL1)" /ren: MyClusterSQL1
This problem is not limited to a SQL Server Resoruce group, and a similar MSDTC problem can be experience in any resource group where the "resource name" of the network name resource is different from the network name itself.