question

AMJ-3470 avatar image
0 Votes"
AMJ-3470 asked MayankBargali-MSFT commented

Logic Apps failed API Get request when using HTTP connector with the message "Could not establish trust relationship for the SSL/TLS secure channel”

When calling Api using HTTP connector (GET method) from Azure Logic App the following message was popped up:

BadRequest. Http request failed with status code 'TrustFailure' and status message: 'The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.'.

The same URL returned correct message in browsers and Postman as below (XML format):

<response status="success">
<result>
<msg>
<line>Report job enqueued with jobid 2334</line>
</msg>
<job>2334</job>
</result>
</response>

The URL did not require authentication. Why did the same URL worked in browsers (and postman) fail to generate a result in HTTP connection of the Logic App? How to overcome the "SSL/TLS" issue in Logic App?

azure-data-factoryazure-logic-apps
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

MayankBargali-MSFT avatar image
1 Vote"
MayankBargali-MSFT answered

Hi @AMJ-3470

Welcome to Microsoft Q&A! Thanks for posting the question.

The error will be observed in the scenario where there is an issue with the certificate at your configured HTTP endpoint or you are using self-sign certificate in multi tenant environment. If you want to use self sign certificate then the logic app should be in ISE. Please review Access for outbound calls to other services and systems for more details on TLS/SSL issue.

In my scenario, I was able to reproduce the issue as the certificate was expired.

Logic App Workflow :

56870-image.png

Error:

56966-image.png

Hope the above helps and you are able to find the cause of the issue by verifying the certificate of your HTTP endpoint.
Feel free to get back to me if you need any assistance.

Please 'Accept as answer' and ‘Upvote’ if it helped so that it can help others in the community looking for help on similar topics.


image.png (20.7 KiB)
image.png (18.9 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

AMJ-3470 avatar image
0 Votes"
AMJ-3470 answered MartinJaffer-MSFT commented

Hi @MayankBargali-MSFT

Thank you so much for your response, really appreciated.

It was neither an issue with the certificate at my configured HTTP endpoint nor I was using a self-sign certificate in multi tenant environment. But I did make an internal api call to my company. That explained the call went through with my browsers/Postman which still got valid SSL/TLS.

ISE is an interesting topic which needs to be injected into VNet in the first place. Unfortunate I was using public cloud upon which PaaS was built. As such I couldn't test out. I may have to ditch logic apps solution to request an xml document.

Instead, I was thinking to use data factory to make a REST api call and save the document id in the data lake with an xml format. Then extract the document id, parse it if required, integrate the document id to the actual url to get the document file back. Any light to shed?

Regards

· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @AMJ-3470

Thanks for your response and sorry to hear that the logic app didn't work for your scenario. Want to confirm that whether you were able to find the cause of the error for SSL/TLS error?
I will check with the data lake team and will get back to you if need more information on your scenario.

0 Votes 0 ·
AMJ-3470 avatar image AMJ-3470 MayankBargali-MSFT ·

Hi @MayankBargali-MSFT,

Please refer to the answer to a separate thread due to written characters allowed in this place. Cheers

0 Votes 0 ·

If you plan to use Data Factory to fetch from your API, you should use the HTTP dataset type, not the REST dataset type.
The reason is that the REST dataset type expects the content to be in JSON format, not XML. The HTTP dataset type can handle XML.

0 Votes 0 ·
AMJ-3470 avatar image
0 Votes"
AMJ-3470 answered MartinJaffer-MSFT commented

Hi @MayankBargali-MSFT,

Very glad to hear from you again. Of course I'd be delighted to find out the cause of SSL/TLS error. So far I had tried

  • Basic authentication

  • Client certificate (with pfx base64 string and password)

  • Managed identity (turn on logic app identity)

All came back with same error message as the one you displayed in your previous message. It seemed all made sense as logic app was in global scope and the url called was in private scope. I even made a data gateway (shared with power bi gateway), but http connector couldn't reference the gateway (all other storage could, such as sql server, sharepoint, data lake). I would think ISE was the only way to succeed. Like I mentioned before there would be lots of overhead in order to use ISE, such as creating VNet, subnets, security groups so on. It would be so grateful if you could confirm what I thought was correct.

On the other hand using data factory needs rest api calls (with self-hosted gateway installed in the first place), which could only return json not xml. Unfortunately my request would return xml (job id), then another request would return xml document. Postman may be another option. It would be great if you could help to get my head around on how to request a private api and deal with xml response. I have been reading MS documents trying to find a way out. So far I do not have a clear direction.

Regards



· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@AMJ-3470 Thanks for sharing the details. You cannot call the API using data gateway in the logic app. You need to create the custom connector that calls the data factory and the data factory calls the on-premises APIs using data gateway. Once the custom connector is created you can call the HTTP API's on your custom connectors from logic app.

0 Votes 0 ·

@MayankBargali-MSFT I've tried custom connector which dealt with either swagger.json or soap xml. In my case the response was rest api xml. So I went back to data factory web api call (GET method) via a on-premises gateway cause the endpoint was a private url. This time I got xml data back as below:

"Response": "<response status=\"success\"><result>\n <msg>\n <line>Report job enqueued with jobid 2525</line>\n </msg>\n <job>2525</job>\n </result></response>"

I need to cut jobId (2525 in the example) out. Because it wasn't json, I couldn't use 'activity'.output.response.result.job to store the value to a variable. I also tried to save the response to either an xml file or a table to sql db, all failed. Any suggestions? Many thanks.

0 Votes 0 ·

@AMJ-3470 , @MayankBargali-MSFT
Since you are using the Web activity, we can write some clever pipeline expression to extract the value. Using your example, you want to extract 2525 which is surrounded by the <job> tag.

 @split(split(pipeline().parameters.input,'</job>')[0],'<job>')[1]


I made a pipeline, with a parameter containing the "Response" you supplied above.
I added a variable and set variable activity with the above expression.

This expression takes the Response string, and splits it using </job> as a separator.
Then it takes the first element of the result, and splits that string using <job> as a separator.
Lastly it takes the second element of this result.

If the "Response" does not contain the job closing tag, then the set variable activity will fail.


0 Votes 0 ·
AMJ-3470 avatar image
0 Votes"
AMJ-3470 answered MartinJaffer-MSFT edited

Hi @MartinJaffer-MSFT,

That was very cool of you to extract out the job id in the example, appreciated.

The question now is how to store the Api response output into a parameter in the first place. Please refer to the attached files.

Eventually the job id cut out will be served as a query parameter to another URL which will return an XML document. So the ultimate question would be how to store the response in a format of xml from a REST Api call in a Web activity to a data lake storage.

From this link (listed below)

https://docs.microsoft.com/en-us/azure/data-factory/control-flow-web-activity

I extracted out "REST endpoints that the web activity invokes must return a response of type JSON. " In other words the response I got back from Web Rest Api call in xml format was useless, could not be processed any further.

All I want is a Rest api call to return an xml document from an on-prem server. I have tried all I could from the list below:

  • Logic app with a gateway (shared with power bi gateway), neither HTTP connector nor Custom Connector would allow me to do so;

  • Azure Data Factory with a separate gateway , neither Web activity nor Rest dataset would take in Rest Api Xml (Rest Api must be in Json format)

  • ISE would require to upgrade my current PaaS to IaaS. There would be lots of overhead and cost could be another concern. So I haven't tried ISE yet.

  • Postman. I could easily get my xml response (job id and document). The question would be how to store the response in Postman to Azure Data Lake?

Please let me know if you had any brainstorm on the matter, gratefully appreciated.


59753-01-api-pipeline.png59670-02-response-output.png



· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

In this case, you would want to use the HTTP connector instead of REST connector. HTTP connector does not have pagination, but it can handle XML and other file formats.
@AMJ-3470 @MayankBargali-MSFT

0 Votes 0 ·
AMJ-3470 avatar image
0 Votes"
AMJ-3470 answered MayankBargali-MSFT commented

Hi @MayankBargali-MSFT,

Custom Connector in Logic App can deal with REST OpenAPI file. In my case it was an on-prem API (not OpenAPI) which could possibly explain why it failed using Custom Connector though a gateway was established.

Then I turned around to ADF, which returned a legit JSON response with XML (job id) embedded in a string as shown in my previous message.

Now the question is 1). How to consume the JSON response requested via WEB activity; or 2). Instead of using WEB activity to request why not use a COPY activity with a REST dataset to save the JSON response to a .json file in a data lake?

All seemed make sense. Prior to using COPY activity I used a JSON validator to make sure the response from WEB activity was valid and it did.

After running the COPY activity I got the message as below:

Failure happened on 'Source' side. ErrorCode=JsonInvalidDataFormat,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Error occurred when deserializing source JSON file ''. Check if the data is in valid JSON object format.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=Newtonsoft.Json.JsonReaderException,Message=Unexpected character encountered while parsing value: <. Path '', line 0, position 0.,Source=Newtonsoft.Json,'

Could you please tell me what it complained about? Was that because the embedded XML string which started with the symbol "<" though it had passed a JSON validator?

Thanks for reminding me to raise a ticket to Microsoft. I do have the permission and authority to raise a ticket to Microsoft as a Chief Azure Data Engineer in my organisation. I'd like to wait until all possible solutions exhausted.




· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @AMJ-3470

Your understanding is correct for the logic app connector.
I will wait for @MartinJaffer-MSFT comment on the ADF side. @MartinJaffer-MSFT Can you help with the ADF queries.

0 Votes 0 ·
AMJ-3470 avatar image AMJ-3470 MayankBargali-MSFT ·

Hi @MayankBargali-MSFT,

Thank you for your genuine help.

At last I've got my XML response from Azure Data Factory and saved XML document to a designate Data Lake. I ditched WEB activity from which the JSON response with embedded XML string could not be consumed and processed any further. I kept using Copy-data activity and altered my linked service from REST to HTTP in the Data Store (inside "File" tab). Problem solved immediately.

I couldn't thank you enough in this journey from which you inspired me a lot. I've learned so much from this little task. Still I couldn't resolve it from Logic App which would motivate me to dig deeper for Azure which resembles a blue sky for people to enjoy.

I am going to close this issue, started up from SSL certificate in Logic App and finished off with HTTP linked service in ADF. What a journey, it was real and fun, may not be a real fun sometimes. So long, mate.

0 Votes 0 ·

@AMJ-3470 Thanks for your kind words and glad to know that the discussion has helped you.
Feel free to get back to us in Microsoft Q&A if you need any help in the future.
It was nice working with you!

0 Votes 0 ·