Issues while Integrating SOAPUI tool with Dynamics CRM 2013 | "HTTP Error 401 – Unauthorized: Access is denied"

  • Article

What is SOAP UI?

SOAPUI tool is a third party tool. It is a quick simple tool to allow you to start testing your SOAP calls with very little effort. I have seen our Microsoft Dynamics CRM users using this a lot and recently I worked with one of my partners where he was using this tool all fine in CRM 2011 until he started using CRM 2013 and tested this tool and it failed with different errors.

I tried looking for information around this tool with regards to Dynamics CRM but found very few blogs and information around it. This is not a Microsoft built tool neither we(Microsoft) support it but I thought sharing by experience on this tool would be worth writing as it might help many of you who are using this tool and if they come across these error. Most of the articles talk about how to use it and all its steps but none mention about the issue you would face while connecting it for the first time.

How it works ?

To start with we grab the CRM .wsdl from our CRM application, which can be downloaded from the ‘Organization Service’ link under Settings---> Customizations ---> Developer Resources. Once we have this we should be ready to create a new SOAP UI project, by consuming the .wsdl we just  downloaded.

Open SOAP UI, and select ‘New Project’ from the File menu. In the ‘New SOAP Project window’, add a project name and open the organization.wsdl you just downloaded.  

In this tool  you will need to create the 3 headers that will be included when using Client side code. We can do this by expanding the Headers section on the bottom of the screen, and then by clicking on the ‘+’ icon to add each of the following headers. (For this call, I have taken example of  EXECUTE, so we will use ‘Execute’ as our action, but this would change depending on the action we want to use) 

Headers:

SOAPAction: (e.g. https://schemas.microsoft.com/xrm/2011/Contracts/Services/IOrganizationService/Execute as we are using execute method)

Content-type: text/xml; charset=utf-8

Accept: application/xml, text/xml, */*

Then we also choose the Auth for it to connect to CRM server. We have 3 options in here  Kerberos, NTLM, Basic. It works fine with NTLM and I have tested it successfully. Things to Remember while using this tool :-

  • This tool does not work with IFD and works on Windows Authentication as far as from what I found during my research.
  • Then URL that we use in this URL, we add “/web” to the end.  If you forget this step, you will most likely get the following error:   

              “The server cannot service the request because the media type is unsupported.”

             e.g.. https://<servername:port>/<orgname>/XRMServices/2011/Organization.svc/web

  • Ideally it should work with Kerberos Authentication but I have seen it successful only in NTLM protocol for windows authentication.

We can use any way to get the SOAP request XML (using Soap Logger etc in CRM SDK).  

 

 

ISSUE

So, once we pasted the xml for SOAP request in SOAP UI tool for Dynamics CRM 2013 the SOAP response XML returned following error to us:- 

"HTTP Error 401 – Unauthorized: Access is denied"

DETAILS and ANALYSIS DONE to FIND THE ROOT CAUSE

 We searched various forum and blogs and found many are complaining about the same error they are hitting using this tool with CRM 2013. 

Note: This blog just mentions my experience on this error and its resolution. This does not mean this would match with very user and every CRM deployment 

We looked at every possible place related to authentication and very setting in our server for authentication, SPN, Kerberos, user password name etc was all good!

 Finally after collecting multiple traces and analysis of NETMON logs we found below:-  

  • We did test using both kerberos and NTLM authentication to connect to CRM 2013, we found that SOAPUI application is not responding to credential challenge (NTLM) by server and thus IIS rejects the request.
  • Ntlm authentication failed because SOAPUI seems to be using Ntlm v1 w\o any encryption which is not default and not recommended. It seems Server hosting CRM 2013 is not configured to respond to Ntlm v1 w\o encryption security and thus authentication fails. 

               - Http: Request, POST /CRMOrgTest/XRMServices/2011/Organization.svc/web , Using NTLM Authorization

                 Command: POST

              + URI: /CRMOrgTest/XRMServices/2011/Organization.svc/web

              + SOAPAction:  https://schemas.microsoft.com/xrm/2011/Contracts/Services/IOrganizationService/Execute

                 UserAgent:  Apache-HttpClient/4.1.1 (java 1.5)

              - Authorization: NTLM

              - Authorization:  NTLM TlRMTVNTUAABAAAAATIAAAMAAwAgAAAACAAIACMAAABDVFNQQzE5NDk2OQ==

              - NTHTNtlmAuthorization:

               Scheme: NTLM

              - NLMP: NTLM NEGOTIATE MESSAGE, Workstation Domain: CTS, Workstation Name: PC194969

                MessageType: Negotiate Message (0x00000001)

             + NegotiateFlags: 0x00003201 (NTLM v1No encryption, No Sign or Seal)

             + DomainNameFields: Length: 3, Offset: 32

             + WorkstationFields: Length: 8, Offset: 35

                WorkstationDomain: CTS

                WorkstationName: PC194969

 

  • Kerberos authentication failed because SOAP UI didn’t respond to credential challenge at all, not sure why! SOAP UI team should look into this and give us some pointers. We checked in network traces and received. We do see that  <<<< Server responds with 401 telling client that it requires credentials and it includes authentication types supported!!, <<<< Next request from Client doesn’t includes Authentication header i.e. credentials are not sent by it, BAD!!
  •  So, logs confirmed that we need to see something from SOAP UI tool itself who is causing failure by either using not allowed authentication protocol or not responding to authentication challenge . To confirm that, we used WCF test client to access /CRMOrgTest/XRMServices/2011/Organization.svc/web endpoint. We found that WCFTestClient was able to pass security successfully! So, it’s confirmed that CRM 2013 Organization.svc/web endpoint is not in fault here!!
  • Then, we worked on a question “Why SOAPUI is able to connect to CRM 2011 Organization.svc/web endpoint?”. Well, based on the network logs for Ntlm, it seems server hosting CRM 2011 allows Ntlm v1 w\o encryption.
  • We did some research to find what configuration on system should allow Ntlm v1 w\o encryption. We found that following security policies could control the behavior

              1.  Network security: LAN Manager authentication level:

                   This security setting determines which challenge/response authentication protocol is used for network

                   logons. This choice affects the level of authentication protocol used by clients, the level of session security

                   negotiated

              2.  Network security: Minimum session security for NTLM SSP based (including secure RPC) clients:

                  This security setting allows a client to require the negotiation of 128-bit encryption and/or NTLMv2 session

                 security. These values are dependent on the LAN Manager Authentication Level security setting value.

              3.  Network security: Minimum session security for NTLM SSP based (including secure RPC) servers:

                  This security setting allows a server to require the negotiation of 128-bit encryption and/or NTLMv2 session

                  security. These values are dependent on the LAN Manager Authentication Level security setting value.

  •   We compared the above security policies on servers hosting CRM 2013 and CRM 2011 and found that Policy# (2) and Policy# (3) above were different on both servers. CRM 2011 machine had set them to Minimum security which will allow Ntlm authentication w\o encryption! 
  •  This confirms why we see different behavior with SOAPUI connecting to CRM 2011 and CRM 2013 Organization.svc/web endpoint.

 

RESOLUTION

 The temporary resolution we used after checking in netmon traces is not a supported one and we should not edit this local security policy to make it work. SOAP UI tool team should look into it and see why it is failing until we do below steps. Microsoft do not support this tool neither recommend to perform below changes. 

  • On the Server opened "Local Security Policy"
  • After matching security policies on both servers (hosting CRM 2011 and CRM 2013) and made below configuration changes on CRM 2013 server that we had in  CRM 2011 server for testing the issue got fixed. i.e.
  1. Opened Local Policies--> Security Policies
  2. Double clicked Network security: Minimum session security for NTLM SSP based (including secure RPC) clients: uncheck "Require 128-bit encryption"
  3. Double clicked Network security: Minimum session security for NTLM SSP based (including secure RPC) servers: uncheck "Require 128-bit encryption"