question

RamamoorthyKesavan-4374 avatar image
0 Votes"
RamamoorthyKesavan-4374 asked ·

Copy Activity(REST source) How to map a particular json key-value to a variable

Hi,

I am using a http connector to pull the data from a rest api source(due to other issues in rest connector) and store the data in ADLS gen2. The source api response is a json file with few fields.

Under 'Mapping' tab in the copy activity dataset, the array 'data' is what we need to store in ADLS and there is no issues in mapping that array to Sink and that has been done already.

The question is, Is there any possibility to assign the values coming from the json output key
"[pagination][cursors][after]" that has been highlighted in the picture to a variable ?

60643-image.png




azure-data-factory
image.png (49.1 KiB)
· 10
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@RamamoorthyKesavan-4374 Thanks for using Microsoft Q&A !!

No, you cannot assign it to a variable in schema mappings. Can you please provide any specific need for doing the same ?

0 Votes 0 ·

Hi @SaurabhSharma-msft ,
Below is the loop that's used to iterate thru' different pages based on pagenumber parameter in api.
60883-image.png


But the people from source team have changed the source api design and there is no more pagenumber parameter in the api and instead they have implemented cursor based approach to iterate thru' pages. During the first api call,response gives us cursorId - "[pagination][cursors][after]" (as shown in schema mapping screenshot earlier) which should be given as api parameter for the second api call. I was thinking like if we capture the [pagination][cursors][after] in some variable, we can use it in the dataset parameter in the same way pagenumber parameter is used today, as shown in another snip. Pls let me know possible approach that can be implementd in loop to iterate based on cursor id which is known only after 1st api call


0 Votes 0 ·
image.png (24.6 KiB)

Hi @SaurabhSharma-msft,
Missed a screenshot in earlier reply. Pls find it below.
60856-image.png


0 Votes 0 ·
image.png (8.8 KiB)
Show more comments

0 Answers