Using HTTP or REST API as source in Copy Activity - error on source

Question

question

MPrice-6021 asked Apr 26, '22 MPrice-6021 commented May 12, '22

Using HTTP or REST API as source in Copy Activity - error on source

I have an Synapse pipeline that will make a call to a REST API returning a json response that I want to write to the data lake. My REST API call requires I pass a sessionId in the header. I first use a WebActivity to 'login' and retrieve the session Id. I then use a Copy Data activity with a HTTP json source. My sink is a json file in the data lake. I have 5 'Additional headers' that I must include when making my API call in the source, two of which reference my sessionId returned from the first Web activity. When I try Preview data on my source I get the following error:

Error occurred when deserializing source JSON file '../applications'. Check if the data is in valid JSON object format.
Unexpected character encountered while parsing value: <. Path '', line 0, position 0

When I run my pipeline it does complete, however the file that is written contains HTML instead of json and appears to be the result of the login failing.

I also tried using a REST API call as my source instead of the HTTP source. Both give me the same error on the Preview data of the source.

To test that my API call is passing the correct values in the header, I created a second pipeline containing a Web activity to 'login' and return the session Id, then I have a second Web activity that passes the 5 header values. This pipeline makes a successful call to my API and returns the expected result.

Here is what my pipeline looks like using an HTTP source with the Copy data activity:

My Additional headers field looks like :
@{concat('Accept: application/json,Content-Type: application/json,Asc_xsrf_token: ',activity('Login').output.sessionId,',Range: items=0-999,Cookie: asc_session_id=',activity('Login').output.sessionId)}

This is what my pipeline looks like when using a REST API source:

And my pipeline containing just a Web Activity to return the response looks like this. The Web activity returns a response as expected:

Am I passing the headers correctly for the Copy data activity? Why would the Copy activity API call not work?
Is there a way to view the Request that is being sent for the source? That would help me debug this issue.

azure-data-factory azure-synapse-analytics azure-data-lake-storage

image.png (45.4 KiB)

image.png (55.5 KiB)

image.png (36.6 KiB)

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Answer 1 · 2022-04-28T14:08:52.2Z

ShaikMaheer-MSFT answered Apr 28, '22 MPrice-6021 commented May 12, '22

Hi @MPrice-6021 ,

Thank you for posting query in Microsoft Q&A Platform.

Could you please try using below string interpolation syntax while writing expression and see if that helps?
Accept: application/json,Content-Type: application/json,Asc_xsrf_token: ',@{activity('Login').output.sessionId},',Range: items=0-999,Cookie: asc_session_id=',@{activity('Login').output.sessionId}

Please let us know how it goes. Thank you.

· 3

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

MPrice-6021 · Apr 28 at 03:11 PM

Thanks for the reply @ShaikMaheer-MSFT ,

I made the change to the expression and get the same error result. I made the suggested change to the pipeline that utilizes an HTTP json source type and made a similar change to the pipeline that uses a REST source type. Both returned the same error I reported initially.

Is there a way to see the actual request that is being sent?

Thank you.

0 ·

ShaikMaheer-MSFT MPrice-6021 · May 02 at 06:52 AM

Hi @MPrice-6021 , If I am not wrong, there is no way to see actual request sent to API. But if input json of that activity can give any clue then we can check it. To double confirm, I am checking with internal team for there inputs if we can see actual request sent to API. I will be share updates once I here back.

0 ·

MPrice-6021 ShaikMaheer-MSFT · May 02 at 03:41 PM

Hi @ShaikMaheer-MSFT ,
Here is the input from Copy Activity when using a REST json source:

The headers look correct to me.

Here is the Input from the Copy Activity when using a HTTP json source:

Assuming the the headers are formatted correctly, this also look correct.

When using a HTTP json source, the Copy Activity does complete without error, however the file written contains the following text instead of a json response:

The Copy Activity that uses a REST json source fails with the error I mentioned in my original question. Based on that error message, I suspect the copy activity is validating the json before writing the file. The HTTP source and the REST source are likely both being redirected to a page for login.

0 ·

image.png (26.7 KiB)

image.png (29.0 KiB)

image.png (26.8 KiB)

question