question

KumariShweta-1759 avatar image
0 Votes"
KumariShweta-1759 asked KumariShweta-1759 commented

How to Fetch 1000 records data from Rest API connect dynamically with Parameters

Hi,

I am trying to fetch 1000 using the rest API connector link services in ADF and storing in ADLS but I am not able to set pagination for the URL.

Example of my URL :https://api.instantwebtools.net/v1/passenger?page=1&size=10

azure-data-factory
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

SathyamoorthyVijayakumar-MSFT avatar image
1 Vote"
SathyamoorthyVijayakumar-MSFT answered KumariShweta-1759 commented

Hello @KumariShwetha-1759

I am assuming that you are making use of the Copy Data activity to move data from REST API to ADLS. If this is not case. Please feel free to provide some context on the activity.

The copy data by default supports the pagination for the below scenarios :

137447-image.png


137370-image.png

Reference : https://docs.microsoft.com/en-us/azure/data-factory/connector-rest?tabs=data-factory#pagination-support

Now, coming back to your scenario

For fetching 1000 records, I am guessing that you would like recursively hit the page like below

 https://api.instantwebtools.net/v1/passenger?page=1&size=10
 https://api.instantwebtools.net/v1/passenger?page=2&size=10
 https://api.instantwebtools.net/v1/passenger?page=3&size=10
 ...
 ...
 ...
 https://api.instantwebtools.net/v1/passenger?page=10&size=10

if the response header/response body have the url for the next page ( for instance, https://api.instantwebtools.net/v1/passenger?page=2&size=10) , you could configure it in the below way.


137409-image.png

If the next page link or query parameter doesn't come as response headers/body then you're less likely to be able to use the feature.

The alternative would be to make use of loop logic & Perform the Copy Activity

In the Rest Connector create two parameters :

137464-image.png

Now populate the relative URL for the RestConnector with the parameters.

137427-image.png

 @concat(dataset().page,dataset().size)

This is to form the Relative URL dynamically. In your case "?page=1&size=10"

Now, In the Copy Activity :

137482-image.png

You could set the page number as a Variable.

 @concat('?page=',variables('page'))

This variable value would be incremented in a loop by making use of the Set Variable activity. for each iteration --> ?page=1, ?page=2,?page=3

The URL for the Copy Activity is dynamically set for each iteration.

To Loop / iterate - You could either make use of Until activity

Condition : Until the variable is less than 100 or so.

After the page 1 is executed - the variable is updated 2.

The request is made with the variable value 2 i.e. the page 2 is queried.

 https://reqres.in/api/users?page=2&size=10

OR

FOREACH Loop (The implementation is discussed here : https://docs.microsoft.com/en-us/answers/questions/59467/how-to-pass-pagination-rules-during-a-rest-api-cal.html)


Note : if needed you could then merge the data for all the files (generated for each page)

Hope this will help. Please let us know if any further queries.


  • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how

  • Want a reminder to come back and check responses? Here is how to subscribe to a notification

  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators



image.png (22.2 KiB)
image.png (69.0 KiB)
image.png (43.4 KiB)
image.png (29.7 KiB)
image.png (18.7 KiB)
image.png (21.0 KiB)
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi Vijay Kumar,

Thank you its really work for me.

0 Votes 0 ·